commit 21e69e06117e06546b09f3ed6032e3e5d18db5f4 parent aaefc14e5bd846015b67febc31b9d928fe114276 Author: ling0x <ling0x@users.noreply.github.com> Date: Mon, 15 Jun 2026 17:09:27 +0100 refactor: txt Diffstat:
146 files changed, 1987 insertions(+), 1949 deletions(-)
diff --git a/AGENTS.md b/AGENTS.txt diff --git a/README.md b/README.txt diff --git a/async_programming/actors.md b/async_programming/actors.txt diff --git a/async_programming/async_futures.md b/async_programming/async_futures.txt diff --git a/async_programming/busy_loop.md b/async_programming/busy_loop.txt diff --git a/async_programming/channels.md b/async_programming/channels.md @@ -1,19 +0,0 @@ -# Channels - -Channels: A [communication primitive](/async_programming/concurrency_primitives.md) -used in many languages (Rust, Go, .NET, etc.) to pass messages between -concurrent pieces of code (goroutines, tasks, threads). - -# Tokio's channel primitives - -- mpsc: multi-producer, single-consumer channel. Many values can be sent. - -- oneshot: single-producer, single consumer channel. A single value can be sent. - -- broadcast: multi-producer, multi-consumer. Many values can be sent. Each - receiver sees every value. - -- watch: multi-producer, multi-consumer. Many values can be sent, but no history - is kept. Receivers only see the most recent value. - - [tokio-tutorial](/exercises/tokio-tutorial/index.md) diff --git a/async_programming/channels.txt b/async_programming/channels.txt @@ -0,0 +1,19 @@ +# Channels + +Channels: A communication primitive +used in many languages (Rust, Go, .NET, etc.) to pass messages between +concurrent pieces of code (goroutines, tasks, threads). + +# Tokio's channel primitives + +- mpsc: multi-producer, single-consumer channel. Many values can be sent. + +- oneshot: single-producer, single consumer channel. A single value can be sent. + +- broadcast: multi-producer, multi-consumer. Many values can be sent. Each + receiver sees every value. + +- watch: multi-producer, multi-consumer. Many values can be sent, but no history + is kept. Receivers only see the most recent value. + + tokio-tutorial diff --git a/async_programming/concurrency_primitives.md b/async_programming/concurrency_primitives.md @@ -1,58 +0,0 @@ -# Concurrency Primitives - -**Concurrency primitives** in Rust are the fundamental building blocks that -allow multiple parts of a program to run simultaneously — safely and without -data races. They are the low-level tools you use to coordinate concurrent tasks, -share data between threads, and synchronize execution. - -## Why Rust Is Special Here - -Rust's ownership and type system enforce concurrency safety at **compile time**, -not at runtime. Many bugs that would silently corrupt data in other languages -become compile errors in Rust, which is why Rust calls its approach "fearless -concurrency". -[doc.rust-lang](https://doc.rust-lang.org/book/ch16-00-concurrency.html) - -## The Core Primitives - -- [**Threads**](/operating_systems/thread.md) — the most basic primitive; - independent paths of execution that run concurrently, letting you exploit - multi-core processors - [earthly](https://earthly.dev/blog/rust-concurrency-patterns-parallel-programming/) -- **Channels (`mpsc`)** — typed message-passing pipes with a sender and receiver - handle; one thread sends data, another receives it, avoiding shared memory - entirely [news.ycombinator](https://news.ycombinator.com/item?id=7851274) -- [**Mutex (`Mutex<T>`)**](/memory_safety/mutex.md) — short for _mutual - exclusion_; only one thread can access the protected data at a time, - preventing data races on shared state - [earthly](https://earthly.dev/blog/rust-concurrency-patterns-parallel-programming/) -- [**Arc (`Arc<T>`)**](/memory_safety/arc.md) — _Atomic Reference Counting_; - lets multiple threads share ownership of a value safely - [doc.rust-lang](https://doc.rust-lang.org/book/ch16-03-shared-state.html) -- [**`RwLock<T>`**](/memory_safety/rwlock_pattern.md) — like a Mutex, but allows - many simultaneous readers or one exclusive writer -- [**Atomic types**](/memory_safety/atomic.md) — low-level primitives (e.g., - `AtomicUsize`) for lock-free, thread-safe operations on simple values - [web.mit](https://web.mit.edu/rust-lang_v1.25/arch/amd64_ubuntu1404/share/doc/rust/html/book/first-edition/concurrency.html) - -## Key Traits: `Send` and `Sync` - -Rust enforces concurrency rules through two marker traits: -[web.mit](https://web.mit.edu/rust-lang_v1.25/arch/amd64_ubuntu1404/share/doc/rust/html/book/first-edition/concurrency.html) - -- **`Send`** — a type can be transferred (moved) to another thread -- **`Sync`** — a type can be safely _referenced_ from multiple threads - simultaneously - -These traits are automatically implemented by the compiler where safe, and -withheld where they aren't — so you can't accidentally send a non-thread-safe -type across a thread boundary. -[google.github](https://google.github.io/comprehensive-rust/concurrency/welcome.html) - -## A Simple Mental Model - -Think of concurrency primitives as traffic rules for threads. Channels say -_"pass the data by handing it off"_, while `Mutex`/`Arc` say _"share the data, -but take turns"_. Rust's compiler acts as the traffic enforcer, rejecting unsafe -patterns before your code ever runs. -[dzone](https://dzone.com/articles/concurrency-in-rust-safe-and-efficient-code) diff --git a/async_programming/concurrency_primitives.txt b/async_programming/concurrency_primitives.txt @@ -0,0 +1,58 @@ +# Concurrency Primitives + +**Concurrency primitives** in Rust are the fundamental building blocks that +allow multiple parts of a program to run simultaneously — safely and without +data races. They are the low-level tools you use to coordinate concurrent tasks, +share data between threads, and synchronize execution. + +## Why Rust Is Special Here + +Rust's ownership and type system enforce concurrency safety at **compile time**, +not at runtime. Many bugs that would silently corrupt data in other languages +become compile errors in Rust, which is why Rust calls its approach "fearless +concurrency". +[doc.rust-lang](https://doc.rust-lang.org/book/ch16-00-concurrency.html) + +## The Core Primitives + +- **Threads** — the most basic primitive; + independent paths of execution that run concurrently, letting you exploit + multi-core processors + [earthly](https://earthly.dev/blog/rust-concurrency-patterns-parallel-programming/) +- **Channels (`mpsc`)** — typed message-passing pipes with a sender and receiver + handle; one thread sends data, another receives it, avoiding shared memory + entirely [news.ycombinator](https://news.ycombinator.com/item?id=7851274) +- **Mutex (`Mutex<T>`)** — short for _mutual + exclusion_; only one thread can access the protected data at a time, + preventing data races on shared state + [earthly](https://earthly.dev/blog/rust-concurrency-patterns-parallel-programming/) +- **Arc (`Arc<T>`)** — _Atomic Reference Counting_; + lets multiple threads share ownership of a value safely + [doc.rust-lang](https://doc.rust-lang.org/book/ch16-03-shared-state.html) +- **`RwLock<T>`** — like a Mutex, but allows + many simultaneous readers or one exclusive writer +- **Atomic types** — low-level primitives (e.g., + `AtomicUsize`) for lock-free, thread-safe operations on simple values + [web.mit](https://web.mit.edu/rust-lang_v1.25/arch/amd64_ubuntu1404/share/doc/rust/html/book/first-edition/concurrency.html) + +## Key Traits: `Send` and `Sync` + +Rust enforces concurrency rules through two marker traits: +[web.mit](https://web.mit.edu/rust-lang_v1.25/arch/amd64_ubuntu1404/share/doc/rust/html/book/first-edition/concurrency.html) + +- **`Send`** — a type can be transferred (moved) to another thread +- **`Sync`** — a type can be safely _referenced_ from multiple threads + simultaneously + +These traits are automatically implemented by the compiler where safe, and +withheld where they aren't — so you can't accidentally send a non-thread-safe +type across a thread boundary. +[google.github](https://google.github.io/comprehensive-rust/concurrency/welcome.html) + +## A Simple Mental Model + +Think of concurrency primitives as traffic rules for threads. Channels say +_"pass the data by handing it off"_, while `Mutex`/`Arc` say _"share the data, +but take turns"_. Rust's compiler acts as the traffic enforcer, rejecting unsafe +patterns before your code ever runs. +[dzone](https://dzone.com/articles/concurrency-in-rust-safe-and-efficient-code) diff --git a/async_programming/cpu.md b/async_programming/cpu.txt diff --git a/async_programming/git_actors_case_study.md b/async_programming/git_actors_case_study.md @@ -1,640 +0,0 @@ -# Git Actors Case Study - -# Why Custom Git Management Uses an Actor System Instead of HTTP Requests - -## Executive Summary - -A real world project uses an actor-based architecture for git management, not -primarily because of WebSocket integration, but because git operations are -inherently stateful, long-running, and require shared mutable state that HTTP's -stateless request-response model cannot efficiently handle. - -See also: -[DAL Architecture Overview](/system_design/dal_architecture_overview.md) - ---- - -## 1. The Core Problem: Stateful Git Operations - -### What HTTP Gives You - -``` -Client → HTTP Request → Server → HTTP Response → Client -``` - -- Stateless: Each request is independent -- Fire-and-forget: No persistent connection -- No shared context between requests - -### What Git Actually Requires - -``` -Client → Load Repo → Edit Files → Save → Commit → Switch Branch → ... - ↓ ↓ ↓ ↓ - [Actor maintains working directory state] -``` - -Git operations are stateful: - -1. You load a repository once -2. Make multiple edits over time -3. The edits persist in a working directory -4. You compile, stage, commit incrementally -5. The state persists until explicitly saved or the session ends - ---- - -## 2. Architectural Analysis of the Codebase - -### The Three-Layer Architecture - -``` -┌──────────────────────────────────────────────────────────────────────────┐ -│ FRONTEND (Svelte) │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ -│ │ FileTree │ │ CodeEditor │ │ GitEditor │ │ Canvas │ │ -│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ -│ │ │ │ │ │ -│ └────────────────┼────────────────┼────────────────┘ │ -│ │ │ -│ ┌───────────▼───────────────┐ │ -│ │ WebSocket Connection │◄──── Persistent Connection │ -│ │ (TypeScript Client) │ │ -│ └─────────────┬─────────────┘ │ -└────────────────────────────┼─────────────────────────────────────────────┘ - │ Binary Protocol (MessagePack) - ▼ -┌─────────────────────────────────────────────────────────────────────────┐ -│ BACKEND (Rust) │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ WebSocket Actor (per-connection) │ │ -│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ -│ │ │ StaticSession │ │ DynamicSession │ │ Ingress Router │ │ │ -│ │ │ (auth, sender) │ │ (actor sender) │ │ │ │ │ -│ │ └────────┬────────┘ └────────┬────────┘ └─────────────────┘ │ │ -│ └───────────┼────────────────────┼────────────────────────────────┘ │ -│ │ │ │ -│ └──────────┬─────────┘ │ -│ │ │ -│ ┌──────────────────────▼──────────────────────────────────────────┐ │ -│ │ ALLOCATOR ACTOR (Singleton per Server) │ │ -│ │ │ │ -│ │ HashMap<(project_id, branch) → (GitActorSender, JoinHandle)> │ │ -│ │ │ │ -│ │ Messages: Register | DeRegister | Kill | GetSender | GC │ │ -│ └─────────────────────────┬───────────────────────────────────────┘ │ -│ │ │ -│ ┌──────────────────┼──────────────────┐ │ -│ ▼ ▼ ▼ │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ -│ │ Git Files │ │ Git Files │ │ Git Files │ ... │ -│ │ Actor 1 │ │ Actor 2 │ │ Actor 3 │ │ -│ │ (proj:1, │ │ (proj:2, │ │ (proj:1, │ │ -│ │ branch:a) │ │ branch:x) │ │ branch:b) │ │ -│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ -│ │ │ │ │ -│ ▼ ▼ ▼ │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ -│ │ Working Dir │ │ Working Dir │ │ Working Dir │ │ -│ │ (TempDir) │ │ (TempDir) │ │ (TempDir) │ │ -│ │ + Compiler │ │ + Compiler │ │ + Compiler │ │ -│ │ State │ │ State │ │ State │ │ -│ └─────────────┘ └─────────────┘ └─────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────┐ │ -│ │ DATABASE (PostgreSQL) │ │ -│ │ ┌────────────┐ ┌────────────┐ ┌────────────────────────┐ │ │ -│ │ │ Git Blobs │ │ Git File │ │ Auth Sessions │ │ │ -│ │ │ (tarballs) │ │ Actor │ │ WebSocket Sessions │ │ │ -│ │ │ │ │ Sessions │ │ │ │ │ -│ │ └────────────┘ └────────────┘ └────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 3. Why Actors, Not HTTP? - -### 3.1 Stateful Working Directory - -```rust - // From git_files/actor.rs - pub async fn git_files_actor<X: ...>( - mut rx: mpsc::Receiver<IncomingFileActorMessage>, - project_id: i32, - branch: String, - Storage handle: Arc<dyn GitDataTransfer + 'static>, - ) -> Result<(), NanoServiceError> { - - // 1. Load git data from storage ONCE - let git_data = storage_handle.load_git_data(project_id, branch.clone()).await?; - - // 2. Unpack to temp directory ONCE - unpack_tar_gz(git_data, source_dir.clone())?; - - // 3. Cache compiler state in memory - let mut compiler_state = EntryPointStates::new(); - let dep_graph = load_from_disk(&source_dir).unwrap_or(DependencyGraph::new()); - - // 4. Listen for operations on this specific working directory - while let Some(message) = rx.recv().await { - match message { - IncomingFileActorMessage::ReadFile(tx, path) => { ... } - IncomingFileActorMessage::WriteFile(tx, path, data) => { ... } - IncomingFileActorMessage::Compile(tx, path) => { ... } - // ... - } - } - } -``` - -If this were HTTP: - -``` -HTTP POST /files/read → Must reload repo, unpack tarball, return file -HTTP POST /files/write → Must reload repo, unpack tarball, write, repack, save -HTTP POST /compile → Must reload repo, unpack tarball, load graph, compile -``` - -Each request would: - -- Download the entire repo from DB (expensive) -- Unpack the tarball (slow) -- Perform tiny operation -- Save back to DB -- No caching of compilation state - -### 3.2 Compiler State Persistence - -```rust - // The actor maintains compilation state across requests - let mut compiler_state = EntryPointStates::new(); - - // First compile: builds dependency graph from scratch - let outcome = compile_entry_point(source_dir.as_path(), file_path, &mut compiler_state).await; - - // Second compile: reuses cached graph, only recompiles changed nodes - let outcome = compile_entry_point(source_dir.as_path(), file_path, &mut compiler_state).await; -``` - -Why this matters: - -- Dependency graphs can be megabytes for complex CAD projects -- Incremental compilation: Change one file → only recompile affected nodes -- HTTP cannot do this: No shared state between requests - -### 3.3 Reference Counting & Session Management - -```rust - // From allocator/actor.rs - pub async fn websocket_allocator_actor(...) { - let mut allocator = AllocatorMap::new(); // In-memory state - - while let Some(message) = rx.recv().await { - match message { - IncomingAllocatorMessage::Register(tx, project_id, branch) => { - // Check if actor already exists - // If yes: increment ref_count, return existing sender - // If no: spawn new actor, return new sender - } - IncomingAllocatorMessage::DeRegister(tx, project_id, branch) => { - // Decrement ref_count - // If ref_count == 0: set time_zeroed for garbage collection - } - } - } - } -``` - -```rust - // From state.rs - pub type AllocatorMap = HashMap<AllocatorKey, (GitActorSender, ActorJoinHandle)>; - - #[derive(Debug, PartialEq, Hash, Eq)] - pub struct AllocatorKey { - pub project_id: i32, - pub branch: String - } -``` - -The Session Model: - -``` -User A opens project 1, branch "main" → ref_count = 1 -User B opens project 1, branch "main" → ref_count = 2, SAME actor -User A closes → ref_count = 1, actor stays alive -User B closes → ref_count = 0, actor marked for GC -``` - -HTTP Alternative Problems: - -``` -HTTP: No persistent state. Each request is independent. -- User A opens project: start session -- User A makes 100 edits: 100 independent requests -- User B opens same project: start ANOTHER session -- Database: Two separate copies of the repo loaded -- Memory: Double memory usage -- Coherence: Two separate working directories, no shared state -``` - -### 3.4 Locking & Consistency - -```rust - // From git_files/actor.rs - AcquireLock message - IncomingFileActorMessage::AcquireLock(sender, rx) => { - let _ = sender.send(OutgoingFileActorMessage::LockAcquired); - - // Mutex-like behavior: hold lock until ReleaseLock - let outcome = match rx.await { - Ok(message) => message, - Err(_) => continue, - }; - match outcome { - IncomingFileActorMessage::ReleaseLock(tx) => { - let _: Result<(), OutgoingFileActorMessage> = - tx.send(OutgoingFileActorMessage::LockReleased); - }, - _ => continue, - } - } -``` - -Why locks matter: - -- Two users editing the same file simultaneously -- User A's write must complete before User B's write -- Actor ensures sequential processing of messages -- No race conditions, no lost updates - -### 3.5 Garbage Collection & Cleanup - -```rust -// From garbage_collector.rs -pub async fn garbage_collector<X>(alloc_sender: AllocatorMessageSender) { - loop { - sleep(Duration::from_secs(20)).await; - let _ = send_gc_request(&alloc_sender).await; - } -} -``` - -The lifecycle: - -1. User opens project → Actor spawned, ref_count = 1 -2. Multiple users open → ref_count incremented -3. User closes → ref_count decremented -4. Last user closes → ref_count = 0, time_zeroed set -5. GC runs every 20 seconds → deletes actors with time_zeroed > 120 seconds ago -6. Grace period: If user reopens within 2 minutes, actor still exists - -HTTP can't do this: No state to clean up, but also no caching benefits. - ---- - -## 4. The WebSocket Integration (Secondary Benefit) - -You asked: "Is it because of its integration with WebSocket frontend?" - -Partially, but it's not the primary reason. Here's the relationship: - -### WebSocket Benefits (Secondary) - -``` -┌───────────────────────────────────────────────────────────────┐ -│ HTTP vs WebSocket │ -├───────────────────────────────────────────────────────────────┤ -│ HTTP: │ -│ - Open connection, send request, get response, close │ -│ - Good for: auth, project CRUD, one-off operations │ -│ │ -│ WebSocket: │ -│ - Persistent connection, bidirectional messaging │ -│ - Good for: real-time file editing, compilation feedback, │ -│ multiplayer sync, server-initiated notifications │ -└───────────────────────────────────────────────────────────────┘ -``` - -### But HTTP Could Also Work with Actors! - -```rust - // Hypothetical HTTP approach with actors (NOT how it's done here) - // HTTP endpoints would still talk to actors internally: - - async fn read_file_handler( - Query((project_id, branch)): Query<(i32, String)>, - ) -> impl IntoResponse { - // 1. Get actor sender from allocator - let sender = get_actor_sender(project_id, &branch).await?; - - // 2. Send message to actor - let path = "file.txt"; - send_read_file_request(path, &sender).await - } -``` - -So why WebSocket? - -| Requirement | HTTP | WebSocket | Actor System | -| ------------------------ | ---- | --------- | ------------ | -| Stateful file operations | ❌ | ⚠️ | ✅ | -| Compiler state caching | ❌ | ❌ | ✅ | -| Session ref counting | ❌ | ⚠️ | ✅ | -| Real-time updates | ❌ | ✅ | ✅ | -| Locking/concurrency | ❌ | ⚠️ | ✅ | -| Server→client push | ❌ | ✅ | ✅ | - -## 5. System Design Patterns Used - -### 5.1 Actor Pattern (Erlang-style) - -```rust -// Single-threaded message processing per actor -while let Some(message) = rx.recv().await { - // Process ONE message at a time - // No locks needed within the actor - // Actor owns all its state -} -``` - -### 5.2 Resource Pool Pattern (Allocator) - -```rust -// One actor per (project, branch) tuple -// Reused across multiple websocket sessions -// Ref counting prevents premature cleanup -``` - -### 5.3 Supervisor Pattern (implied) - -```rust -// If actor panics, it's isolated -// WebSocket handler aborts ping actor -// Cleanup still runs on session end -``` - -### 5.4 Message Passing Concurrency - -```rust -// No shared mutable state -// Communication via channels only -// Type-safe message protocols -``` - -### 5.5 Event Sourcing (implied) - -```rust -// Changes don't modify stored data immediately -// SaveSnapshot packages entire working dir -// Stored as tarball in DB (immutable blob) -``` - ---- - -## 6. Comparison: Actor vs HTTP Architectures - -### Actor-Based (Current) - -``` -┌────────────────────────────────────────────────────────────────────┐ -│ ACTOR SYSTEM │ -├───────┬───┬──────────────┬───────────────┬───────────────┬─────────┤ -│ │ │ -│ Session 1 ─┬─► DynamicSession ─┬─► Allocator ──┬─► GitActor ──────┼──► TempDir -│ │ │ │ │ + State -│ Session 2 ─┤ │ │ │ -│ │ │ └─► GitActor ──────┼──► TempDir -│ Session 3 ─┘ │ │ + State -│ │ │ -│ Global GC ────────────────────────────────────────────────────────┼──► Cleanup -│ │ -│ ✓ Single copy of repo in memory per (project, branch) │ -│ ✓ Compiler state persists across requests │ -│ ✓ Atomic operations with locks │ -│ ✓ Graceful cleanup via ref counting │ -│ ✓ WebSocket naturally maps to actor sessions │ -└────────────────────────────────────────────────────────────────────┘ -``` - -### HTTP-Based Alternative (Theoretical) - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ HTTP STATELESS SYSTEM │ -├───────┬───┬──────────────┬───────────────┬───────────────┬──────┤ -│ │ │ -│ HTTP/1 ──► Load Project ──► Edit ──► Save ──► Compile ──► ... │ -│ │ │ │ │ │ -│ ▼ ▼ ▼ ▼ │ -│ Download Upload Upload Upload │ -│ tarball tarball tarball tarball │ -│ │ │ │ │ │ -│ └───────────┴────────┴─┬─────┘ │ -│ DB (every operation) │ -│ │ -│ ✗ Download entire repo on every request │ -│ ✗ Unpack/repack tarball on every operation │ -│ ✗ No compiler state caching │ -│ ✗ No incremental compilation │ -│ ✗ Multiple users = multiple copies of same repo │ -│ ✗ Slow response times │ -│ ✗ High database load │ -└─────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 7. Summary: Why Actor System Wins - -| Factor | HTTP | Actor System | -| ------------------------ | ------------------------------ | ------------------------------- | -| Stateful file operations | ❌ Would need external cache | ✅ Actors own working directory | -| Compiler state | ❌ Must reload on each compile | ✅ Cached in memory | -| Incremental compilation | ❌ Full rebuild every time | ✅ Only changed nodes | -| Session management | ⚠️ External session store | ✅ Ref counting built-in | -| Locking | ⚠️ Database locks | ✅ Message queue serializes | -| Memory efficiency | ❌ N copies for N users | ✅ 1 copy shared via sender | -| Real-time updates | ❌ Long-polling/comet | ✅ WebSocket native | -| Graceful cleanup | ⚠️ TTL-based | ✅ GC with grace period | -| WebSocket integration | ⚠️ Request-response over WS | ✅ Native message passing | -| Multiplayer sync | ❌ Complex broadcast logic | ✅ Actors as session boundaries | - -The actor system is used because git operations are fundamentally stateful, and -the actor model provides: - -1. Stateful working directories — Load once, edit many times -2. Compiler state caching — Incremental compilation via dependency graphs -3. Session multiplexing — One actor, many websocket connections -4. Reference counting — Memory-efficient session management -5. Locking — Sequential - ---- - -## 9. Reference: HashMap Allocator Details - -### What the HashMap Allocator Does for the Git Management Actor - -The AllocatorMap (`HashMap<AllocatorKey, (GitActorSender, ActorJoinHandle)>`) -acts as the in-memory registry for all running git file actors on this server. -Here's what it does: - -#### Core Responsibilities - -1. **Tracks Live Actors**: Every time a new git file actor is spawned (on first - Register), its (sender, join_handle) tuple gets inserted into the HashMap - keyed by (project_id, branch). -2. **Enables Actor Reuse**: When a subsequent Register comes in for the same - project/branch, instead of spawning a new actor, the allocator: - - Increments the ref_count in the DB (for tracking how many clients are using - it) - - Looks up the existing sender in the HashMap and returns it (no new actor - spawned) - - This is why both senders in your multi-client tests work — they point to - the same actor. -3. **Provides Fast O(1) Sender Lookup**: The `get_sender` process does a HashMap - lookup to retrieve a cloned sender. This is a synchronous, non-DB operation — - critical for low-latency websocket routing. -4. **Cleans Up on Server Restart**: The actor starts by wiping all sessions in - the DB for this server tag, ensuring stale state from a previous crashed - instance is gone. -5. **Enables Targeted Kill**: The kill process removes the entry from the - HashMap, then waits on the join handle to confirm the actor stopped. - -#### Relationship Between Websocket Actor Allocator and Git File Actor Allocator - -They are the same allocator — **there is only one actor managing everything**. -Here's how they relate: - -``` -┌────────────────────────────────────────────────────────────────────┐ -│ WEBSOCKET CONNECTION #1 │ -│ (one per connected client browser) │ -│ │ -│ - Owns its own ping actor (health monitoring) │ -│ - Owns DynamicSession + StaticSession state │ -│ - Communicates with the allocator via mpsc channel │ -└────────────────────────────────────────────────────────────────────┘ - │ - │ send_register_request() - ▼ -┌────────────────────────────────────────────────────────────────────┐ -│ ALLOCATOR ACTOR (Single Global Actor) │ -│ ┌───────────────────────────────────────────────────────────────┐ │ -│ │ AllocatorMap HashMap (in-memory state) │ │ -│ │ Key: AllocatorKey { project_id, branch } │ │ -│ │ Value: (GitActorSender, JoinHandle) │ │ -│ │ │ │ -│ │ Example entries: │ │ -│ │ (project:42, "main") -> (sender_A, handle_1) │ │ -│ │ (project:42, "dev") -> (sender_B, handle_2) │ │ -│ │ (project:99, "main") -> (sender_C, handle_3) │ │ -│ └───────────────────────────────────────────────────────────────┘ │ -│ │ -│ Receives messages: Register, DeRegister, Kill, GetSender, GC │ -└────────────────────────────────────────────────────────────────────┘ - │ - ┌─────────────┴────────────┐ - │ │ - ▼ ▼ -┌──────────────────────────────────┐ ┌──────────────────────────────────┐ -│ GIT FILE ACTOR │ │ GIT FILE ACTOR │ -│ (project:42, branch:"main") │ │ (project:42, branch:"dev") │ -│ │ │ │ -│ - Owns temp dir on disk │ │ - Owns temp dir on disk │ -│ - Handles file operations │ │ - Handles file operations │ -│ - Has compiler state │ │ - Has compiler state │ -│ - Persists to DB on save │ │ - Persists to DB on save │ -└──────────────────────────────────┘ └──────────────────────────────────┘ -``` - -#### The Message Flow - -``` -Client Browser Websocket Actor - │ │ - │──── websocket connect ─────────────────>│ - │ │ - │ │ 1. auth check - │ │ 2. calls allocator_actor_constructor() - │ │ (static singleton, runs once per server) - │ │ - │ │ 3. send_register_request(project_id, "main") - │ │ to allocator via mpsc::Sender - │ │ - │ ▼ - │ ┌───────────────────┐ - │ │ ALLOCATOR ACTOR │ - │ │ │ - │ │ checks DB → no existing session - │ │ spawns git_files_actor_constructor() - │ │ inserts into HashMap - │ │ creates DB session (ref_count=1) - │ └───────────────────┘ - │ │ - │ │ returns GitActorSender - │ │ - │ ▼ - │ ┌───────────────────┐ - │ │ GIT FILE ACTOR │ - │ │ (project:42, main)│ - │ │ - loads tarball │ - │ │ - extracts files │ - │ │ - in-memory state │ - │ └───────────────────┘ - │ │ - │<────── websocket messages ──────────────┤ - │ (routed to git file actor) │ - │ │ - │ ─────── disconnect ────────────────────>│ - │ │ cleanup() → send_deregister_request() - │ │ ref_count decremented in DB - │ │ HashMap entry NOT removed (actor stays alive) - │ │ - │ [if ref_count == 0, background GC sends Kill - │ → removes from HashMap, deletes DB session] -``` - -#### Key Distinction: Two Different Session Types - -| Session Type | Storage | Purpose | Managed By | -| ---------------------- | ---------------------------------- | -------------------------------------------------------------------- | ------------------------------------ | -| Websocket Session | DB only (websocket_sessions table) | Track which users are connected, server tag for cleanup | Websocket actor's cleanup() function | -| Git File Actor Session | DB + HashMap | Track actor lifecycle, ref_count for sharing, server tag for cleanup | Allocator actor | - -The websocket session exists purely in the DB to survive server restarts (so you -know a user was connected before the crash). The git file actor session lives in -both DB and HashMap — the DB for persistence across restarts, the HashMap for -fast in-memory access. - -#### Why One Allocator Handles Both - -The name "websocket allocator" in some comments is a bit misleading — it -**doesn't allocate websocket connections**. It allocates git file actors on -behalf of websocket connections. All websocket connections on the server share -the same allocator actor because: - -1. **Actor Model**: The allocator is a single tokio task with an mpsc channel. - All websocket actors send messages to it. -2. **Shared State**: The HashMap needs to be shared across all websocket - connections so they can find/reuse the same git file actor for a given - project/branch. -3. **Efficiency**: One allocator means one place to manage actor lifecycles, - reference counting, and cleanup — avoiding distributed state synchronization - issues. - -#### The Reference Counting Mechanism - -``` -Register → ref_count++ -DeRegister → ref_count-- (DB sets time_zeroed when it hits 0) - ↓ - Background GC polls DB every 30s - ↓ - Finds sessions where time_zeroed IS NOT NULL - ↓ - Sends Kill to allocator → removes from HashMap, deletes DB row -``` - -This is how the system handles clients disconnecting at different times — the -actor only dies when the last client deregisters. diff --git a/async_programming/git_actors_case_study.txt b/async_programming/git_actors_case_study.txt @@ -0,0 +1,640 @@ +# Git Actors Case Study + +# Why Custom Git Management Uses an Actor System Instead of HTTP Requests + +## Executive Summary + +A real world project uses an actor-based architecture for git management, not +primarily because of WebSocket integration, but because git operations are +inherently stateful, long-running, and require shared mutable state that HTTP's +stateless request-response model cannot efficiently handle. + +See also: +DAL Architecture Overview + +--- + +## 1. The Core Problem: Stateful Git Operations + +### What HTTP Gives You + +``` +Client → HTTP Request → Server → HTTP Response → Client +``` + +- Stateless: Each request is independent +- Fire-and-forget: No persistent connection +- No shared context between requests + +### What Git Actually Requires + +``` +Client → Load Repo → Edit Files → Save → Commit → Switch Branch → ... + ↓ ↓ ↓ ↓ + [Actor maintains working directory state] +``` + +Git operations are stateful: + +1. You load a repository once +2. Make multiple edits over time +3. The edits persist in a working directory +4. You compile, stage, commit incrementally +5. The state persists until explicitly saved or the session ends + +--- + +## 2. Architectural Analysis of the Codebase + +### The Three-Layer Architecture + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ FRONTEND (Svelte) │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ FileTree │ │ CodeEditor │ │ GitEditor │ │ Canvas │ │ +│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ +│ │ │ │ │ │ +│ └────────────────┼────────────────┼────────────────┘ │ +│ │ │ +│ ┌───────────▼───────────────┐ │ +│ │ WebSocket Connection │◄──── Persistent Connection │ +│ │ (TypeScript Client) │ │ +│ └─────────────┬─────────────┘ │ +└────────────────────────────┼─────────────────────────────────────────────┘ + │ Binary Protocol (MessagePack) + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ BACKEND (Rust) │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ WebSocket Actor (per-connection) │ │ +│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ +│ │ │ StaticSession │ │ DynamicSession │ │ Ingress Router │ │ │ +│ │ │ (auth, sender) │ │ (actor sender) │ │ │ │ │ +│ │ └────────┬────────┘ └────────┬────────┘ └─────────────────┘ │ │ +│ └───────────┼────────────────────┼────────────────────────────────┘ │ +│ │ │ │ +│ └──────────┬─────────┘ │ +│ │ │ +│ ┌──────────────────────▼──────────────────────────────────────────┐ │ +│ │ ALLOCATOR ACTOR (Singleton per Server) │ │ +│ │ │ │ +│ │ HashMap<(project_id, branch) → (GitActorSender, JoinHandle)> │ │ +│ │ │ │ +│ │ Messages: Register | DeRegister | Kill | GetSender | GC │ │ +│ └─────────────────────────┬───────────────────────────────────────┘ │ +│ │ │ +│ ┌──────────────────┼──────────────────┐ │ +│ ▼ ▼ ▼ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Git Files │ │ Git Files │ │ Git Files │ ... │ +│ │ Actor 1 │ │ Actor 2 │ │ Actor 3 │ │ +│ │ (proj:1, │ │ (proj:2, │ │ (proj:1, │ │ +│ │ branch:a) │ │ branch:x) │ │ branch:b) │ │ +│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Working Dir │ │ Working Dir │ │ Working Dir │ │ +│ │ (TempDir) │ │ (TempDir) │ │ (TempDir) │ │ +│ │ + Compiler │ │ + Compiler │ │ + Compiler │ │ +│ │ State │ │ State │ │ State │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────┐ │ +│ │ DATABASE (PostgreSQL) │ │ +│ │ ┌────────────┐ ┌────────────┐ ┌────────────────────────┐ │ │ +│ │ │ Git Blobs │ │ Git File │ │ Auth Sessions │ │ │ +│ │ │ (tarballs) │ │ Actor │ │ WebSocket Sessions │ │ │ +│ │ │ │ │ Sessions │ │ │ │ │ +│ │ └────────────┘ └────────────┘ └────────────────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## 3. Why Actors, Not HTTP? + +### 3.1 Stateful Working Directory + +```rust + // From git_files/actor.rs + pub async fn git_files_actor<X: ...>( + mut rx: mpsc::Receiver<IncomingFileActorMessage>, + project_id: i32, + branch: String, + Storage handle: Arc<dyn GitDataTransfer + 'static>, + ) -> Result<(), NanoServiceError> { + + // 1. Load git data from storage ONCE + let git_data = storage_handle.load_git_data(project_id, branch.clone()).await?; + + // 2. Unpack to temp directory ONCE + unpack_tar_gz(git_data, source_dir.clone())?; + + // 3. Cache compiler state in memory + let mut compiler_state = EntryPointStates::new(); + let dep_graph = load_from_disk(&source_dir).unwrap_or(DependencyGraph::new()); + + // 4. Listen for operations on this specific working directory + while let Some(message) = rx.recv().await { + match message { + IncomingFileActorMessage::ReadFile(tx, path) => { ... } + IncomingFileActorMessage::WriteFile(tx, path, data) => { ... } + IncomingFileActorMessage::Compile(tx, path) => { ... } + // ... + } + } + } +``` + +If this were HTTP: + +``` +HTTP POST /files/read → Must reload repo, unpack tarball, return file +HTTP POST /files/write → Must reload repo, unpack tarball, write, repack, save +HTTP POST /compile → Must reload repo, unpack tarball, load graph, compile +``` + +Each request would: + +- Download the entire repo from DB (expensive) +- Unpack the tarball (slow) +- Perform tiny operation +- Save back to DB +- No caching of compilation state + +### 3.2 Compiler State Persistence + +```rust + // The actor maintains compilation state across requests + let mut compiler_state = EntryPointStates::new(); + + // First compile: builds dependency graph from scratch + let outcome = compile_entry_point(source_dir.as_path(), file_path, &mut compiler_state).await; + + // Second compile: reuses cached graph, only recompiles changed nodes + let outcome = compile_entry_point(source_dir.as_path(), file_path, &mut compiler_state).await; +``` + +Why this matters: + +- Dependency graphs can be megabytes for complex CAD projects +- Incremental compilation: Change one file → only recompile affected nodes +- HTTP cannot do this: No shared state between requests + +### 3.3 Reference Counting & Session Management + +```rust + // From allocator/actor.rs + pub async fn websocket_allocator_actor(...) { + let mut allocator = AllocatorMap::new(); // In-memory state + + while let Some(message) = rx.recv().await { + match message { + IncomingAllocatorMessage::Register(tx, project_id, branch) => { + // Check if actor already exists + // If yes: increment ref_count, return existing sender + // If no: spawn new actor, return new sender + } + IncomingAllocatorMessage::DeRegister(tx, project_id, branch) => { + // Decrement ref_count + // If ref_count == 0: set time_zeroed for garbage collection + } + } + } + } +``` + +```rust + // From state.rs + pub type AllocatorMap = HashMap<AllocatorKey, (GitActorSender, ActorJoinHandle)>; + + #[derive(Debug, PartialEq, Hash, Eq)] + pub struct AllocatorKey { + pub project_id: i32, + pub branch: String + } +``` + +The Session Model: + +``` +User A opens project 1, branch "main" → ref_count = 1 +User B opens project 1, branch "main" → ref_count = 2, SAME actor +User A closes → ref_count = 1, actor stays alive +User B closes → ref_count = 0, actor marked for GC +``` + +HTTP Alternative Problems: + +``` +HTTP: No persistent state. Each request is independent. +- User A opens project: start session +- User A makes 100 edits: 100 independent requests +- User B opens same project: start ANOTHER session +- Database: Two separate copies of the repo loaded +- Memory: Double memory usage +- Coherence: Two separate working directories, no shared state +``` + +### 3.4 Locking & Consistency + +```rust + // From git_files/actor.rs - AcquireLock message + IncomingFileActorMessage::AcquireLock(sender, rx) => { + let _ = sender.send(OutgoingFileActorMessage::LockAcquired); + + // Mutex-like behavior: hold lock until ReleaseLock + let outcome = match rx.await { + Ok(message) => message, + Err(_) => continue, + }; + match outcome { + IncomingFileActorMessage::ReleaseLock(tx) => { + let _: Result<(), OutgoingFileActorMessage> = + tx.send(OutgoingFileActorMessage::LockReleased); + }, + _ => continue, + } + } +``` + +Why locks matter: + +- Two users editing the same file simultaneously +- User A's write must complete before User B's write +- Actor ensures sequential processing of messages +- No race conditions, no lost updates + +### 3.5 Garbage Collection & Cleanup + +```rust +// From garbage_collector.rs +pub async fn garbage_collector<X>(alloc_sender: AllocatorMessageSender) { + loop { + sleep(Duration::from_secs(20)).await; + let _ = send_gc_request(&alloc_sender).await; + } +} +``` + +The lifecycle: + +1. User opens project → Actor spawned, ref_count = 1 +2. Multiple users open → ref_count incremented +3. User closes → ref_count decremented +4. Last user closes → ref_count = 0, time_zeroed set +5. GC runs every 20 seconds → deletes actors with time_zeroed > 120 seconds ago +6. Grace period: If user reopens within 2 minutes, actor still exists + +HTTP can't do this: No state to clean up, but also no caching benefits. + +--- + +## 4. The WebSocket Integration (Secondary Benefit) + +You asked: "Is it because of its integration with WebSocket frontend?" + +Partially, but it's not the primary reason. Here's the relationship: + +### WebSocket Benefits (Secondary) + +``` +┌───────────────────────────────────────────────────────────────┐ +│ HTTP vs WebSocket │ +├───────────────────────────────────────────────────────────────┤ +│ HTTP: │ +│ - Open connection, send request, get response, close │ +│ - Good for: auth, project CRUD, one-off operations │ +│ │ +│ WebSocket: │ +│ - Persistent connection, bidirectional messaging │ +│ - Good for: real-time file editing, compilation feedback, │ +│ multiplayer sync, server-initiated notifications │ +└───────────────────────────────────────────────────────────────┘ +``` + +### But HTTP Could Also Work with Actors! + +```rust + // Hypothetical HTTP approach with actors (NOT how it's done here) + // HTTP endpoints would still talk to actors internally: + + async fn read_file_handler( + Query((project_id, branch)): Query<(i32, String)>, + ) -> impl IntoResponse { + // 1. Get actor sender from allocator + let sender = get_actor_sender(project_id, &branch).await?; + + // 2. Send message to actor + let path = "file.txt"; + send_read_file_request(path, &sender).await + } +``` + +So why WebSocket? + +| Requirement | HTTP | WebSocket | Actor System | +| ------------------------ | ---- | --------- | ------------ | +| Stateful file operations | ❌ | ⚠️ | ✅ | +| Compiler state caching | ❌ | ❌ | ✅ | +| Session ref counting | ❌ | ⚠️ | ✅ | +| Real-time updates | ❌ | ✅ | ✅ | +| Locking/concurrency | ❌ | ⚠️ | ✅ | +| Server→client push | ❌ | ✅ | ✅ | + +## 5. System Design Patterns Used + +### 5.1 Actor Pattern (Erlang-style) + +```rust +// Single-threaded message processing per actor +while let Some(message) = rx.recv().await { + // Process ONE message at a time + // No locks needed within the actor + // Actor owns all its state +} +``` + +### 5.2 Resource Pool Pattern (Allocator) + +```rust +// One actor per (project, branch) tuple +// Reused across multiple websocket sessions +// Ref counting prevents premature cleanup +``` + +### 5.3 Supervisor Pattern (implied) + +```rust +// If actor panics, it's isolated +// WebSocket handler aborts ping actor +// Cleanup still runs on session end +``` + +### 5.4 Message Passing Concurrency + +```rust +// No shared mutable state +// Communication via channels only +// Type-safe message protocols +``` + +### 5.5 Event Sourcing (implied) + +```rust +// Changes don't modify stored data immediately +// SaveSnapshot packages entire working dir +// Stored as tarball in DB (immutable blob) +``` + +--- + +## 6. Comparison: Actor vs HTTP Architectures + +### Actor-Based (Current) + +``` +┌────────────────────────────────────────────────────────────────────┐ +│ ACTOR SYSTEM │ +├───────┬───┬──────────────┬───────────────┬───────────────┬─────────┤ +│ │ │ +│ Session 1 ─┬─► DynamicSession ─┬─► Allocator ──┬─► GitActor ──────┼──► TempDir +│ │ │ │ │ + State +│ Session 2 ─┤ │ │ │ +│ │ │ └─► GitActor ──────┼──► TempDir +│ Session 3 ─┘ │ │ + State +│ │ │ +│ Global GC ────────────────────────────────────────────────────────┼──► Cleanup +│ │ +│ ✓ Single copy of repo in memory per (project, branch) │ +│ ✓ Compiler state persists across requests │ +│ ✓ Atomic operations with locks │ +│ ✓ Graceful cleanup via ref counting │ +│ ✓ WebSocket naturally maps to actor sessions │ +└────────────────────────────────────────────────────────────────────┘ +``` + +### HTTP-Based Alternative (Theoretical) + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ HTTP STATELESS SYSTEM │ +├───────┬───┬──────────────┬───────────────┬───────────────┬──────┤ +│ │ │ +│ HTTP/1 ──► Load Project ──► Edit ──► Save ──► Compile ──► ... │ +│ │ │ │ │ │ +│ ▼ ▼ ▼ ▼ │ +│ Download Upload Upload Upload │ +│ tarball tarball tarball tarball │ +│ │ │ │ │ │ +│ └───────────┴────────┴─┬─────┘ │ +│ DB (every operation) │ +│ │ +│ ✗ Download entire repo on every request │ +│ ✗ Unpack/repack tarball on every operation │ +│ ✗ No compiler state caching │ +│ ✗ No incremental compilation │ +│ ✗ Multiple users = multiple copies of same repo │ +│ ✗ Slow response times │ +│ ✗ High database load │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## 7. Summary: Why Actor System Wins + +| Factor | HTTP | Actor System | +| ------------------------ | ------------------------------ | ------------------------------- | +| Stateful file operations | ❌ Would need external cache | ✅ Actors own working directory | +| Compiler state | ❌ Must reload on each compile | ✅ Cached in memory | +| Incremental compilation | ❌ Full rebuild every time | ✅ Only changed nodes | +| Session management | ⚠️ External session store | ✅ Ref counting built-in | +| Locking | ⚠️ Database locks | ✅ Message queue serializes | +| Memory efficiency | ❌ N copies for N users | ✅ 1 copy shared via sender | +| Real-time updates | ❌ Long-polling/comet | ✅ WebSocket native | +| Graceful cleanup | ⚠️ TTL-based | ✅ GC with grace period | +| WebSocket integration | ⚠️ Request-response over WS | ✅ Native message passing | +| Multiplayer sync | ❌ Complex broadcast logic | ✅ Actors as session boundaries | + +The actor system is used because git operations are fundamentally stateful, and +the actor model provides: + +1. Stateful working directories — Load once, edit many times +2. Compiler state caching — Incremental compilation via dependency graphs +3. Session multiplexing — One actor, many websocket connections +4. Reference counting — Memory-efficient session management +5. Locking — Sequential + +--- + +## 9. Reference: HashMap Allocator Details + +### What the HashMap Allocator Does for the Git Management Actor + +The AllocatorMap (`HashMap<AllocatorKey, (GitActorSender, ActorJoinHandle)>`) +acts as the in-memory registry for all running git file actors on this server. +Here's what it does: + +#### Core Responsibilities + +1. **Tracks Live Actors**: Every time a new git file actor is spawned (on first + Register), its (sender, join_handle) tuple gets inserted into the HashMap + keyed by (project_id, branch). +2. **Enables Actor Reuse**: When a subsequent Register comes in for the same + project/branch, instead of spawning a new actor, the allocator: + - Increments the ref_count in the DB (for tracking how many clients are using + it) + - Looks up the existing sender in the HashMap and returns it (no new actor + spawned) + - This is why both senders in your multi-client tests work — they point to + the same actor. +3. **Provides Fast O(1) Sender Lookup**: The `get_sender` process does a HashMap + lookup to retrieve a cloned sender. This is a synchronous, non-DB operation — + critical for low-latency websocket routing. +4. **Cleans Up on Server Restart**: The actor starts by wiping all sessions in + the DB for this server tag, ensuring stale state from a previous crashed + instance is gone. +5. **Enables Targeted Kill**: The kill process removes the entry from the + HashMap, then waits on the join handle to confirm the actor stopped. + +#### Relationship Between Websocket Actor Allocator and Git File Actor Allocator + +They are the same allocator — **there is only one actor managing everything**. +Here's how they relate: + +``` +┌────────────────────────────────────────────────────────────────────┐ +│ WEBSOCKET CONNECTION #1 │ +│ (one per connected client browser) │ +│ │ +│ - Owns its own ping actor (health monitoring) │ +│ - Owns DynamicSession + StaticSession state │ +│ - Communicates with the allocator via mpsc channel │ +└────────────────────────────────────────────────────────────────────┘ + │ + │ send_register_request() + ▼ +┌────────────────────────────────────────────────────────────────────┐ +│ ALLOCATOR ACTOR (Single Global Actor) │ +│ ┌───────────────────────────────────────────────────────────────┐ │ +│ │ AllocatorMap HashMap (in-memory state) │ │ +│ │ Key: AllocatorKey { project_id, branch } │ │ +│ │ Value: (GitActorSender, JoinHandle) │ │ +│ │ │ │ +│ │ Example entries: │ │ +│ │ (project:42, "main") -> (sender_A, handle_1) │ │ +│ │ (project:42, "dev") -> (sender_B, handle_2) │ │ +│ │ (project:99, "main") -> (sender_C, handle_3) │ │ +│ └───────────────────────────────────────────────────────────────┘ │ +│ │ +│ Receives messages: Register, DeRegister, Kill, GetSender, GC │ +└────────────────────────────────────────────────────────────────────┘ + │ + ┌─────────────┴────────────┐ + │ │ + ▼ ▼ +┌──────────────────────────────────┐ ┌──────────────────────────────────┐ +│ GIT FILE ACTOR │ │ GIT FILE ACTOR │ +│ (project:42, branch:"main") │ │ (project:42, branch:"dev") │ +│ │ │ │ +│ - Owns temp dir on disk │ │ - Owns temp dir on disk │ +│ - Handles file operations │ │ - Handles file operations │ +│ - Has compiler state │ │ - Has compiler state │ +│ - Persists to DB on save │ │ - Persists to DB on save │ +└──────────────────────────────────┘ └──────────────────────────────────┘ +``` + +#### The Message Flow + +``` +Client Browser Websocket Actor + │ │ + │──── websocket connect ─────────────────>│ + │ │ + │ │ 1. auth check + │ │ 2. calls allocator_actor_constructor() + │ │ (static singleton, runs once per server) + │ │ + │ │ 3. send_register_request(project_id, "main") + │ │ to allocator via mpsc::Sender + │ │ + │ ▼ + │ ┌───────────────────┐ + │ │ ALLOCATOR ACTOR │ + │ │ │ + │ │ checks DB → no existing session + │ │ spawns git_files_actor_constructor() + │ │ inserts into HashMap + │ │ creates DB session (ref_count=1) + │ └───────────────────┘ + │ │ + │ │ returns GitActorSender + │ │ + │ ▼ + │ ┌───────────────────┐ + │ │ GIT FILE ACTOR │ + │ │ (project:42, main)│ + │ │ - loads tarball │ + │ │ - extracts files │ + │ │ - in-memory state │ + │ └───────────────────┘ + │ │ + │<────── websocket messages ──────────────┤ + │ (routed to git file actor) │ + │ │ + │ ─────── disconnect ────────────────────>│ + │ │ cleanup() → send_deregister_request() + │ │ ref_count decremented in DB + │ │ HashMap entry NOT removed (actor stays alive) + │ │ + │ [if ref_count == 0, background GC sends Kill + │ → removes from HashMap, deletes DB session] +``` + +#### Key Distinction: Two Different Session Types + +| Session Type | Storage | Purpose | Managed By | +| ---------------------- | ---------------------------------- | -------------------------------------------------------------------- | ------------------------------------ | +| Websocket Session | DB only (websocket_sessions table) | Track which users are connected, server tag for cleanup | Websocket actor's cleanup() function | +| Git File Actor Session | DB + HashMap | Track actor lifecycle, ref_count for sharing, server tag for cleanup | Allocator actor | + +The websocket session exists purely in the DB to survive server restarts (so you +know a user was connected before the crash). The git file actor session lives in +both DB and HashMap — the DB for persistence across restarts, the HashMap for +fast in-memory access. + +#### Why One Allocator Handles Both + +The name "websocket allocator" in some comments is a bit misleading — it +**doesn't allocate websocket connections**. It allocates git file actors on +behalf of websocket connections. All websocket connections on the server share +the same allocator actor because: + +1. **Actor Model**: The allocator is a single tokio task with an mpsc channel. + All websocket actors send messages to it. +2. **Shared State**: The HashMap needs to be shared across all websocket + connections so they can find/reuse the same git file actor for a given + project/branch. +3. **Efficiency**: One allocator means one place to manage actor lifecycles, + reference counting, and cleanup — avoiding distributed state synchronization + issues. + +#### The Reference Counting Mechanism + +``` +Register → ref_count++ +DeRegister → ref_count-- (DB sets time_zeroed when it hits 0) + ↓ + Background GC polls DB every 30s + ↓ + Finds sessions where time_zeroed IS NOT NULL + ↓ + Sends Kill to allocator → removes from HashMap, deletes DB row +``` + +This is how the system handles clients disconnecting at different times — the +actor only dies when the last client deregisters. diff --git a/async_programming/queuing.md b/async_programming/queuing.txt diff --git a/async_programming/send_and_sync.md b/async_programming/send_and_sync.txt diff --git a/async_programming/synchronization_primitives.md b/async_programming/synchronization_primitives.md @@ -1,10 +0,0 @@ -# Synchronization Primitives - -[“Concurrency primitives”](/async_programming/concurrency_primitives.md) is the -broader term: it means basic building blocks for concurrent programming, -including things like threads/tasks, queues, futures, and also synchronization -mechanisms. - -“Synchronization primitives” is a subset that specifically deals with ordering -and coordinated access to shared state (e.g. mutexes, rwlocks, semaphores, -barriers, condition variables, atomics. diff --git a/async_programming/synchronization_primitives.txt b/async_programming/synchronization_primitives.txt @@ -0,0 +1,10 @@ +# Synchronization Primitives + +“Concurrency primitives” is the +broader term: it means basic building blocks for concurrent programming, +including things like threads/tasks, queues, futures, and also synchronization +mechanisms. + +“Synchronization primitives” is a subset that specifically deals with ordering +and coordinated access to shared state (e.g. mutexes, rwlocks, semaphores, +barriers, condition variables, atomics. diff --git a/compiler/compiler.md b/compiler/compiler.txt diff --git a/compiler/compiler_tokens.md b/compiler/compiler_tokens.txt diff --git a/compiler/define_dal_transactions_macro.md b/compiler/define_dal_transactions_macro.txt diff --git a/compiler/dynamic_dispatch.md b/compiler/dynamic_dispatch.md @@ -1,116 +0,0 @@ -# Dynamic Dispatch - -**Dynamic dispatch** is when you have multiple types that something might be, -but you don't know which one until runtime. So you **dynamically** figure out -which one of the methods to call on the type. Usually doing this requires the -use of **dyn** trait. - -Dynamic dispatch is when the concrete type implementing a trait is resolved at -**runtime** rather than compile time. In Rust, this is done via trait objects -(`dyn Trait`), and the compiler implements it using a -[**vtable**](/compiler/vtable.md) — a table of function pointers generated for -each concrete type. - -## [Actors](/async_programming/actors.md) are dynamic dispatch - -When you use actors, you don't need dynamic dispatch, because actors provide -dynamic dispatch on theri own. - -Storing something in an actor is an alternative to `Box<dyn Trait>`. - -```rust -struct MyActor<T: AsyncRead + AsyncWrite> { - receiver: mpsc::Receiver<ActorMessage>, - connection: T, -} -``` - -Here, the generic T doesn't leak to the sender. - -```rust -#[derive(Clone)] -pub struct MyActorHandle { - sender: mpsc::Sender<ActorMessage>, -} -``` - -Here, T could be TcpStream or UnixStream depending on the connection. Remote -connection uses TcpStream, whereas local connection uses UnixStream, for -example. - -## Static vs Dynamic Dispatch - -In **static dispatch**, the compiler monomorphizes generic functions — it -generates a separate copy of the function for each concrete type used. Calls are -resolved at compile time and can be inlined. - -```rust -fn area<T: Shape>(s: &T) -> f64 { - s.area() // resolved at compile time -} -``` - -In **dynamic dispatch**, you use a trait object. The compiler doesn't know the -concrete type at compile time, so it emits a vtable lookup at each call site. - -```rust -fn area(s: &dyn Shape) -> f64 { - s.area() // resolved at runtime via vtable -} -``` - -## How the Compiler Implements It - -A `dyn Trait` value is a **fat pointer** — two machine words: - -- A **data pointer** to the value itself -- A **vtable pointer** to a static table of function pointers for the concrete - type - -The vtable is emitted by the compiler for each `(ConcreteType, Trait)` pair. -When you call a method on a `dyn Trait`, Rust loads the function pointer from -the vtable and calls it indirectly. This means no inlining and a small overhead, -but it enables **heterogeneous collections** and **type erasure**. - -``` -Box<dyn Shape> -├── data ptr ──► [ Circle { radius: 3.0 } ] -└── vtable ptr ─► [ drop, size, align, area, ... ] -``` - -## Object Safety - -Not every trait can be used as `dyn Trait`. A trait must be **object-safe** for -this. The key rules are: - -- Methods must not return `Self` -- Methods must not have generic type parameters -- The trait must not require `Sized` - -The compiler enforces this — if a trait is not object-safe, using `dyn Trait` is -a compile error. - -## When to Use Dynamic Dispatch - -Prefer dynamic dispatch when: - -- You need a heterogeneous collection (e.g. `Vec<Box<dyn Plugin>>`) -- You want to hide a concrete type behind an abstraction boundary (e.g. plugin - systems, renderers, handlers) -- Binary size matters more than the last bit of performance (monomorphization - bloat is real) - -Prefer static dispatch when call-site performance and inlining are critical, or -when the set of concrete types is small and known ahead of time. - -## Relationship to Ownership - -Dynamic dispatch is orthogonal to both concurrency and memory safety. You can -wrap a trait object in any ownership primitive: - -- `Box<dyn Trait>` — heap-allocated, single owner -- `Rc<dyn Trait>` — reference-counted, single-threaded -- `Arc<dyn Trait + Send + Sync>` — reference-counted, multi-threaded - -The ownership wrapper controls lifetime and thread safety; the vtable only -controls how methods are dispatched. diff --git a/compiler/dynamic_dispatch.txt b/compiler/dynamic_dispatch.txt @@ -0,0 +1,116 @@ +# Dynamic Dispatch + +**Dynamic dispatch** is when you have multiple types that something might be, +but you don't know which one until runtime. So you **dynamically** figure out +which one of the methods to call on the type. Usually doing this requires the +use of **dyn** trait. + +Dynamic dispatch is when the concrete type implementing a trait is resolved at +**runtime** rather than compile time. In Rust, this is done via trait objects +(`dyn Trait`), and the compiler implements it using a +**vtable** — a table of function pointers generated for +each concrete type. + +## Actors are dynamic dispatch + +When you use actors, you don't need dynamic dispatch, because actors provide +dynamic dispatch on theri own. + +Storing something in an actor is an alternative to `Box<dyn Trait>`. + +```rust +struct MyActor<T: AsyncRead + AsyncWrite> { + receiver: mpsc::Receiver<ActorMessage>, + connection: T, +} +``` + +Here, the generic T doesn't leak to the sender. + +```rust +#[derive(Clone)] +pub struct MyActorHandle { + sender: mpsc::Sender<ActorMessage>, +} +``` + +Here, T could be TcpStream or UnixStream depending on the connection. Remote +connection uses TcpStream, whereas local connection uses UnixStream, for +example. + +## Static vs Dynamic Dispatch + +In **static dispatch**, the compiler monomorphizes generic functions — it +generates a separate copy of the function for each concrete type used. Calls are +resolved at compile time and can be inlined. + +```rust +fn area<T: Shape>(s: &T) -> f64 { + s.area() // resolved at compile time +} +``` + +In **dynamic dispatch**, you use a trait object. The compiler doesn't know the +concrete type at compile time, so it emits a vtable lookup at each call site. + +```rust +fn area(s: &dyn Shape) -> f64 { + s.area() // resolved at runtime via vtable +} +``` + +## How the Compiler Implements It + +A `dyn Trait` value is a **fat pointer** — two machine words: + +- A **data pointer** to the value itself +- A **vtable pointer** to a static table of function pointers for the concrete + type + +The vtable is emitted by the compiler for each `(ConcreteType, Trait)` pair. +When you call a method on a `dyn Trait`, Rust loads the function pointer from +the vtable and calls it indirectly. This means no inlining and a small overhead, +but it enables **heterogeneous collections** and **type erasure**. + +``` +Box<dyn Shape> +├── data ptr ──► [ Circle { radius: 3.0 } ] +└── vtable ptr ─► [ drop, size, align, area, ... ] +``` + +## Object Safety + +Not every trait can be used as `dyn Trait`. A trait must be **object-safe** for +this. The key rules are: + +- Methods must not return `Self` +- Methods must not have generic type parameters +- The trait must not require `Sized` + +The compiler enforces this — if a trait is not object-safe, using `dyn Trait` is +a compile error. + +## When to Use Dynamic Dispatch + +Prefer dynamic dispatch when: + +- You need a heterogeneous collection (e.g. `Vec<Box<dyn Plugin>>`) +- You want to hide a concrete type behind an abstraction boundary (e.g. plugin + systems, renderers, handlers) +- Binary size matters more than the last bit of performance (monomorphization + bloat is real) + +Prefer static dispatch when call-site performance and inlining are critical, or +when the set of concrete types is small and known ahead of time. + +## Relationship to Ownership + +Dynamic dispatch is orthogonal to both concurrency and memory safety. You can +wrap a trait object in any ownership primitive: + +- `Box<dyn Trait>` — heap-allocated, single owner +- `Rc<dyn Trait>` — reference-counted, single-threaded +- `Arc<dyn Trait + Send + Sync>` — reference-counted, multi-threaded + +The ownership wrapper controls lifetime and thread safety; the vtable only +controls how methods are dispatched. diff --git a/compiler/dynamic_libraries.md b/compiler/dynamic_libraries.md @@ -1,268 +0,0 @@ -# Dynamic Libraries - -A **dynamic library** is a compiled binary that is loaded into a program's -address space at either load time or runtime, rather than being copied into the -final executable by the linker. On Linux these files typically have a `.so` -extension (shared object), on Windows `.dll` (dynamic-link library), and on -macOS `.dylib` (dynamic library). - -Dynamic libraries enable **code sharing** between multiple processes: the -operating system maps a single physical copy of the library into the virtual -address space of every process that needs it. This reduces total memory -footprint and allows a library to be updated independently of the executables -that depend on it. - -## Dynamic vs. Static Libraries - -| Feature | Static Library | Dynamic Library | -| :------ | :------------- | :---------------- | -| Linking phase | Compile / link time | Load time or runtime | -| Binary size | Larger executable (library copied in) | Smaller executable (reference only) | -| Memory sharing | Each process gets its own copy | OS shares one physical copy | -| Updates | Recompile executable to update library | Replace library file, restart process | -| Portability | Self-contained executable | Requires compatible library present at runtime | - -A static library (`.a` on Unix, `.lib` on Windows) is essentially an archive of -object files. The linker extracts the needed object files and copies them into -the final executable. Once linked, the static library is no longer needed to run -the program. - -A dynamic library remains a separate file on disk. The executable contains a -**reference** to the library — typically a recorded name and a symbol table of -needed functions — and the operating system's dynamic loader resolves that -reference when the process starts (load-time linking) or when the program -explicitly requests it (runtime linking). - -## How the OS Loads Dynamic Libraries - -When a program starts, the operating system's dynamic loader (e.g., `ld.so` on -Linux, `dyld` on macOS, or the Windows loader) inspects the executable's -**interpreter** and **dynamic section** to determine which shared libraries are -required. It performs the following steps: - -1. **Dependency resolution** — read the list of needed libraries from the - executable headers. -2. **Library search** — locate each library on the search path (`LD_LIBRARY_PATH`, - system cache `/etc/ld.so.cache`, `rpath`, `runpath`, or default system - directories). -3. **Loading and mapping** — `mmap` the library into the process's address space. -4. **Symbol resolution** — walk the relocation tables and patch addresses so - that function calls in the executable point to the correct offsets in the - loaded library. -5. **Initialization** — run constructor functions (e.g., `__attribute__((constructor))` - in C) registered in the library. - -This process is known as **dynamic linking**. If a required library cannot be -found, the loader aborts and the program fails to start. - -## Runtime Loading with `dlopen` and Friends - -Programs can also load libraries explicitly after they have already started. -This is **runtime dynamic linking** and is the mechanism behind plugin systems, -extensible applications, and language interpreters that load native extensions. - -On POSIX systems the C standard library provides four key functions: - -- `dlopen(path, flags)` — load a shared object into the current address space -- `dlsym(handle, symbol)` — retrieve the address of a named symbol (function or - variable) -- `dlclose(handle)` — decrement the reference count and possibly unload the library -- `dlerror()` — return a human-readable string describing the last error - -On Windows the analogous APIs are `LoadLibraryA`, `GetProcAddress`, and -`FreeLibrary`. - -### Example: A Minimal Dynamic C Library - -Imagine a small C library that computes a checksum. Save this as `checksum.c`: - -```c -#include <stdint.h> - -uint32_t checksum(const uint8_t *data, size_t len) { - uint32_t sum = 0; - for (size_t i = 0; i < len; i++) { - sum = (sum << 1) | (sum >> 31); // rotate left - sum += data[i]; - } - return sum; -} -``` - -Compile it into a shared object on Linux: - -```bash -gcc -shared -fPIC -o libchecksum.so checksum.c -``` - -The `-fPIC` flag tells the compiler to emit **position-independent code** — -machine code that can execute correctly regardless of where in memory it is -mapped. This is mandatory for shared libraries because the OS may load them at -different base addresses in different processes (or in the same process across -restarts) for security reasons such as ASLR (Address Space Layout Randomization). - -## Loading a Dynamic C Library from Rust - -Rust can interact with dynamic C libraries through two mechanisms: - -1. **Compile-time dynamic linking** — declare `#[link(name = "checksum")]` and - let the Rust linker record a dependency on `libchecksum.so`. The OS loader - resolves it automatically when the program starts. -2. **Runtime dynamic loading** — use a crate such as `libloading` to `dlopen` - the library manually and look up symbols on demand. - -Runtime loading is more flexible because the program can decide at execution time -whether to load a library, handle failures gracefully, and even swap -implementations without restarting. - -### Using `libloading` - -Add `libloading` to your `Cargo.toml`: - -```toml -[dependencies] -libloading = "0.8" -``` - -Then load the library and call its function: - -```rust -use libloading::{Library, Symbol}; -use std::ffi::c_void; - -type ChecksumFn = unsafe extern "C" fn(*const u8, usize) -> u32; - -fn main() -> Result<(), Box<dyn std::error::Error>> { - unsafe { - let lib = Library::new("./libchecksum.so")?; - - let checksum: Symbol<ChecksumFn> = lib.get(b"checksum\0")?; - - let data = b"hello dynamic world"; - let result = checksum(data.as_ptr(), data.len()); - - println!("Checksum: {}", result); - } - - Ok(()) -} -``` - -Key observations about this code: - -- The `unsafe` block is required because the compiler cannot verify the - correctness of a C library's ABI, pointer usage, or thread safety. -- The symbol name `b"checksum\0"` is a null-terminated byte string, matching the - C ABI expectation. -- `Library::new` calls `dlopen` (or `LoadLibrary` on Windows) under the hood. -- `Symbol` is essentially a smart pointer to a function loaded from a - [**vtable**](/compiler/vtable.md)-like structure inside the dynamic loader. - -### Safety and ABI Mismatches - -When calling into a dynamic C library, Rust's usual memory-safety guarantees do -not apply across the FFI boundary. The C library operates on raw pointers and -expects a specific **application binary interface (ABI)**. If the Rust side -misdeclares a function signature — for example, using `u64` instead of `usize`, -or omitting `extern "C"` — the behavior is undefined and may result in a -[segfault](/memory_safety/segfault.md) or silent data corruption. - -Best practices for dynamic C library interop: - -- Define a thin, audited FFI module that mirrors the C headers exactly. -- Use `std::os::raw` or the `libc` crate for C types (`c_int`, `c_char`, etc.). -- Keep `unsafe` blocks as small as possible; validate inputs before crossing the - boundary. -- Never pass Rust references (`&T`) directly to C code expecting mutable access - unless you have proven alias safety manually. - -## Relocation and the Global Offset Table - -Because a shared library can be loaded at any address, it cannot contain -absolute addresses for its own functions or global data. Instead, the compiler -emits **position-independent code** that refers to a **Global Offset Table -(GOT)** and a **Procedure Linkage Table (PLT)**. - -- **GOT** — an array of pointers to global data. The code reads data indirectly - through the GOT so that only the table entries need to be patched at load time, - not every instruction that references the data. -- **PLT** — a trampoline for external function calls. The first time a function - is called through the PLT, the dynamic linker resolves the real address and - patches the GOT entry; subsequent calls jump directly to the resolved target. - -This lazy resolution (often called **lazy binding**) improves startup time for -large programs with many shared libraries because not every symbol needs to be -resolved immediately. - -## Rust as a Dynamic Library Producer - -Rust can also produce dynamic libraries for consumption by other languages. Two -crate types are relevant: - -- **`cdylib`** — produces a C-compatible dynamic library (`.so`, `.dll`, `.dylib`). - Use this when a C program (or Python via ctypes, or another language) needs to - load your Rust code dynamically. The Rust compiler strips Rust-specific metadata - and exports only functions marked `#[no_mangle]` and `pub extern "C"`. -- **`dylib`** — produces a Rust-native dynamic library. This is primarily used - for rustc plugins and is rarely appropriate for general FFI. - -Example `Cargo.toml` for a C-callable Rust library: - -```toml -[package] -name = "rust_checksum" -version = "0.1.0" -edition = "2021" - -[lib] -crate-type = ["cdylib"] -``` - -And the corresponding Rust source: - -```rust -#[no_mangle] -pub extern "C" fn rust_checksum(data: *const u8, len: usize) -> u32 { - if data.is_null() { - return 0; - } - let slice = unsafe { std::slice::from_raw_parts(data, len) }; - slice.iter().fold(0u32, |acc, &b| acc.wrapping_mul(31).wrapping_add(b as u32)) -} -``` - -Other languages can now `dlopen` the resulting `.so` and call `rust_checksum` -just as if it were a C function. - -## When to Use Dynamic Libraries - -Prefer dynamic libraries when: - -- Multiple executables share the same code and you want to reduce disk and - memory usage. -- You need a plugin architecture where third-party code is loaded at runtime. -- You want to update a library (e.g., a security patch) without rebuilding every - dependent executable. -- You are shipping a large framework and want consumers to link against a stable - ABI without caring about your internal implementation details. - -Prefer static linking when: - -- You need a self-contained binary that runs without external dependencies - (e.g., containers, embedded systems, or CLI tools distributed to unknown - environments). -- Maximum startup performance is critical and you want to avoid symbol - resolution overhead. -- You want whole-program optimization (LTO) to inline across library boundaries. - -## Summary - -Dynamic libraries are separate binaries loaded by the OS loader either at -process startup or on demand via `dlopen` / `LoadLibrary`. They reduce memory -usage through sharing and enable runtime extensibility, but they introduce -complexities around symbol resolution, ABI stability, and distribution. - -When loading a dynamic C library from Rust, you cross an **FFI boundary** where -the compiler's usual safety checks no longer apply. Tools like `libloading` -make runtime loading ergonomic, but correctness depends on matching the C ABI -exactly and carefully auditing every `unsafe` call site. diff --git a/compiler/dynamic_libraries.txt b/compiler/dynamic_libraries.txt @@ -0,0 +1,268 @@ +# Dynamic Libraries + +A **dynamic library** is a compiled binary that is loaded into a program's +address space at either load time or runtime, rather than being copied into the +final executable by the linker. On Linux these files typically have a `.so` +extension (shared object), on Windows `.dll` (dynamic-link library), and on +macOS `.dylib` (dynamic library). + +Dynamic libraries enable **code sharing** between multiple processes: the +operating system maps a single physical copy of the library into the virtual +address space of every process that needs it. This reduces total memory +footprint and allows a library to be updated independently of the executables +that depend on it. + +## Dynamic vs. Static Libraries + +| Feature | Static Library | Dynamic Library | +| :------ | :------------- | :---------------- | +| Linking phase | Compile / link time | Load time or runtime | +| Binary size | Larger executable (library copied in) | Smaller executable (reference only) | +| Memory sharing | Each process gets its own copy | OS shares one physical copy | +| Updates | Recompile executable to update library | Replace library file, restart process | +| Portability | Self-contained executable | Requires compatible library present at runtime | + +A static library (`.a` on Unix, `.lib` on Windows) is essentially an archive of +object files. The linker extracts the needed object files and copies them into +the final executable. Once linked, the static library is no longer needed to run +the program. + +A dynamic library remains a separate file on disk. The executable contains a +**reference** to the library — typically a recorded name and a symbol table of +needed functions — and the operating system's dynamic loader resolves that +reference when the process starts (load-time linking) or when the program +explicitly requests it (runtime linking). + +## How the OS Loads Dynamic Libraries + +When a program starts, the operating system's dynamic loader (e.g., `ld.so` on +Linux, `dyld` on macOS, or the Windows loader) inspects the executable's +**interpreter** and **dynamic section** to determine which shared libraries are +required. It performs the following steps: + +1. **Dependency resolution** — read the list of needed libraries from the + executable headers. +2. **Library search** — locate each library on the search path (`LD_LIBRARY_PATH`, + system cache `/etc/ld.so.cache`, `rpath`, `runpath`, or default system + directories). +3. **Loading and mapping** — `mmap` the library into the process's address space. +4. **Symbol resolution** — walk the relocation tables and patch addresses so + that function calls in the executable point to the correct offsets in the + loaded library. +5. **Initialization** — run constructor functions (e.g., `__attribute__((constructor))` + in C) registered in the library. + +This process is known as **dynamic linking**. If a required library cannot be +found, the loader aborts and the program fails to start. + +## Runtime Loading with `dlopen` and Friends + +Programs can also load libraries explicitly after they have already started. +This is **runtime dynamic linking** and is the mechanism behind plugin systems, +extensible applications, and language interpreters that load native extensions. + +On POSIX systems the C standard library provides four key functions: + +- `dlopen(path, flags)` — load a shared object into the current address space +- `dlsym(handle, symbol)` — retrieve the address of a named symbol (function or + variable) +- `dlclose(handle)` — decrement the reference count and possibly unload the library +- `dlerror()` — return a human-readable string describing the last error + +On Windows the analogous APIs are `LoadLibraryA`, `GetProcAddress`, and +`FreeLibrary`. + +### Example: A Minimal Dynamic C Library + +Imagine a small C library that computes a checksum. Save this as `checksum.c`: + +```c +#include <stdint.h> + +uint32_t checksum(const uint8_t *data, size_t len) { + uint32_t sum = 0; + for (size_t i = 0; i < len; i++) { + sum = (sum << 1) | (sum >> 31); // rotate left + sum += data[i]; + } + return sum; +} +``` + +Compile it into a shared object on Linux: + +```bash +gcc -shared -fPIC -o libchecksum.so checksum.c +``` + +The `-fPIC` flag tells the compiler to emit **position-independent code** — +machine code that can execute correctly regardless of where in memory it is +mapped. This is mandatory for shared libraries because the OS may load them at +different base addresses in different processes (or in the same process across +restarts) for security reasons such as ASLR (Address Space Layout Randomization). + +## Loading a Dynamic C Library from Rust + +Rust can interact with dynamic C libraries through two mechanisms: + +1. **Compile-time dynamic linking** — declare `#[link(name = "checksum")]` and + let the Rust linker record a dependency on `libchecksum.so`. The OS loader + resolves it automatically when the program starts. +2. **Runtime dynamic loading** — use a crate such as `libloading` to `dlopen` + the library manually and look up symbols on demand. + +Runtime loading is more flexible because the program can decide at execution time +whether to load a library, handle failures gracefully, and even swap +implementations without restarting. + +### Using `libloading` + +Add `libloading` to your `Cargo.toml`: + +```toml +[dependencies] +libloading = "0.8" +``` + +Then load the library and call its function: + +```rust +use libloading::{Library, Symbol}; +use std::ffi::c_void; + +type ChecksumFn = unsafe extern "C" fn(*const u8, usize) -> u32; + +fn main() -> Result<(), Box<dyn std::error::Error>> { + unsafe { + let lib = Library::new("./libchecksum.so")?; + + let checksum: Symbol<ChecksumFn> = lib.get(b"checksum\0")?; + + let data = b"hello dynamic world"; + let result = checksum(data.as_ptr(), data.len()); + + println!("Checksum: {}", result); + } + + Ok(()) +} +``` + +Key observations about this code: + +- The `unsafe` block is required because the compiler cannot verify the + correctness of a C library's ABI, pointer usage, or thread safety. +- The symbol name `b"checksum\0"` is a null-terminated byte string, matching the + C ABI expectation. +- `Library::new` calls `dlopen` (or `LoadLibrary` on Windows) under the hood. +- `Symbol` is essentially a smart pointer to a function loaded from a + **vtable**-like structure inside the dynamic loader. + +### Safety and ABI Mismatches + +When calling into a dynamic C library, Rust's usual memory-safety guarantees do +not apply across the FFI boundary. The C library operates on raw pointers and +expects a specific **application binary interface (ABI)**. If the Rust side +misdeclares a function signature — for example, using `u64` instead of `usize`, +or omitting `extern "C"` — the behavior is undefined and may result in a +segfault or silent data corruption. + +Best practices for dynamic C library interop: + +- Define a thin, audited FFI module that mirrors the C headers exactly. +- Use `std::os::raw` or the `libc` crate for C types (`c_int`, `c_char`, etc.). +- Keep `unsafe` blocks as small as possible; validate inputs before crossing the + boundary. +- Never pass Rust references (`&T`) directly to C code expecting mutable access + unless you have proven alias safety manually. + +## Relocation and the Global Offset Table + +Because a shared library can be loaded at any address, it cannot contain +absolute addresses for its own functions or global data. Instead, the compiler +emits **position-independent code** that refers to a **Global Offset Table +(GOT)** and a **Procedure Linkage Table (PLT)**. + +- **GOT** — an array of pointers to global data. The code reads data indirectly + through the GOT so that only the table entries need to be patched at load time, + not every instruction that references the data. +- **PLT** — a trampoline for external function calls. The first time a function + is called through the PLT, the dynamic linker resolves the real address and + patches the GOT entry; subsequent calls jump directly to the resolved target. + +This lazy resolution (often called **lazy binding**) improves startup time for +large programs with many shared libraries because not every symbol needs to be +resolved immediately. + +## Rust as a Dynamic Library Producer + +Rust can also produce dynamic libraries for consumption by other languages. Two +crate types are relevant: + +- **`cdylib`** — produces a C-compatible dynamic library (`.so`, `.dll`, `.dylib`). + Use this when a C program (or Python via ctypes, or another language) needs to + load your Rust code dynamically. The Rust compiler strips Rust-specific metadata + and exports only functions marked `#[no_mangle]` and `pub extern "C"`. +- **`dylib`** — produces a Rust-native dynamic library. This is primarily used + for rustc plugins and is rarely appropriate for general FFI. + +Example `Cargo.toml` for a C-callable Rust library: + +```toml +[package] +name = "rust_checksum" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] +``` + +And the corresponding Rust source: + +```rust +#[no_mangle] +pub extern "C" fn rust_checksum(data: *const u8, len: usize) -> u32 { + if data.is_null() { + return 0; + } + let slice = unsafe { std::slice::from_raw_parts(data, len) }; + slice.iter().fold(0u32, |acc, &b| acc.wrapping_mul(31).wrapping_add(b as u32)) +} +``` + +Other languages can now `dlopen` the resulting `.so` and call `rust_checksum` +just as if it were a C function. + +## When to Use Dynamic Libraries + +Prefer dynamic libraries when: + +- Multiple executables share the same code and you want to reduce disk and + memory usage. +- You need a plugin architecture where third-party code is loaded at runtime. +- You want to update a library (e.g., a security patch) without rebuilding every + dependent executable. +- You are shipping a large framework and want consumers to link against a stable + ABI without caring about your internal implementation details. + +Prefer static linking when: + +- You need a self-contained binary that runs without external dependencies + (e.g., containers, embedded systems, or CLI tools distributed to unknown + environments). +- Maximum startup performance is critical and you want to avoid symbol + resolution overhead. +- You want whole-program optimization (LTO) to inline across library boundaries. + +## Summary + +Dynamic libraries are separate binaries loaded by the OS loader either at +process startup or on demand via `dlopen` / `LoadLibrary`. They reduce memory +usage through sharing and enable runtime extensibility, but they introduce +complexities around symbol resolution, ABI stability, and distribution. + +When loading a dynamic C library from Rust, you cross an **FFI boundary** where +the compiler's usual safety checks no longer apply. Tools like `libloading` +make runtime loading ergonomic, but correctness depends on matching the C ABI +exactly and carefully auditing every `unsafe` call site. diff --git a/compiler/lifetime.md b/compiler/lifetime.txt diff --git a/compiler/non_lexical_lifetimes.md b/compiler/non_lexical_lifetimes.txt diff --git a/compiler/vtable.md b/compiler/vtable.txt diff --git a/engineering_practice/software-entropy.md b/engineering_practice/software-entropy.txt diff --git a/engineering_practice/spotify_model.md b/engineering_practice/spotify_model.txt diff --git a/food/chinese-tomato-chicken-soup.md b/food/chinese-tomato-chicken-soup.txt diff --git a/food/cinnamon-ginger-tea.md b/food/cinnamon-ginger-tea.txt diff --git a/food/coffee_and_cacao_comparison.md b/food/coffee_and_cacao_comparison.txt diff --git a/food/dragon-breath-chai.md b/food/dragon-breath-chai.txt diff --git a/food/garlic-antibiotic-resistance-and-infections.md b/food/garlic-antibiotic-resistance-and-infections.txt diff --git a/food/lemony-chicken-green-olives.md b/food/lemony-chicken-green-olives.txt diff --git a/food/liver-cakes.md b/food/liver-cakes.txt diff --git a/food/nutrients.md b/food/nutrients.txt diff --git a/food/pan-fried-king-prawns.md b/food/pan-fried-king-prawns.txt diff --git a/food/salmon-muffins.md b/food/salmon-muffins.txt diff --git a/linear_algebra/vectors/dot_product.txt b/linear_algebra/vectors/dot_product.txt @@ -106,4 +106,6 @@ Positivity: = 4 / (sqrt(5) * sqrt(5)) = 4/5 - θ = arccos(4/5) -\ No newline at end of file + θ = arccos(4/5) + + Reference read for arccos: https://www.math.net/arccos +\ No newline at end of file diff --git a/machine_learning/concepts.md b/machine_learning/concepts.txt diff --git a/machine_learning/variational_autoencoders.md b/machine_learning/variational_autoencoders.txt diff --git a/memory_safety/Static Functions with Traits and Generics vs. Arc and Instance Methods in Rust.md b/memory_management/Static Functions with Traits and Generics vs. Arc and Instance Methods in Rust.txt diff --git a/memory_safety/arc.md b/memory_management/arc.txt diff --git a/memory_safety/atomic.md b/memory_management/atomic.txt diff --git a/memory_management/borrowing_rules.txt b/memory_management/borrowing_rules.txt @@ -0,0 +1,71 @@ +# Borrowing Rules + +Rust enforces two hard rules at compile time: + +1. You can have any number of immutable references (&T) at the same time + +2. You can have only one mutable reference (&mut T) at a time — and when a + mutable reference is active, no immutable references may exist either + +### String example: + +#### Question: Why does rust only allow one muatble reference at a time? + +A String in Rust is heap-allocated and stores a pointer, a length, and a +capacity internally. If you hold an immutable reference r1 to a String and then +mutate it via a second mutable reference r2 — say by pushing characters — the +String may reallocate its internal buffer to a new heap address. At that point, +r1 would be pointing to freed memory, which is a dangling pointer and causes. +Rust prevents this entirely at compile time. + +```rust +let mut s = String::from("hello"); + +let r1 = &s; // ✅ immutable borrow +let r2 = &s; // ✅ another immutable borrow — fine! +let r3 = &mut s; // ❌ compile error: can't borrow mutably while r1/r2 exist +``` + +### Furthermore: + +The single mutable reference rule actually solves several classes of bugs +simultaneously: + +- Dangling pointers — mutation causes reallocation, invalidating old references + (your example) + +- Data races — two threads mutating the same memory simultaneously leads to + unpredictable results + +- Iterator invalidation — modifying a collection + while iterating over it (a common bug in C++ and Java) + +- Compiler optimisation safety — the compiler can safely optimise and even + vectorise (SIMD) code because it knows no two mutable aliases can overlap + +## Mutable Reference Lifetime Is Scoped + +The borrow checker is smart enough to track when a reference's last use is, not +just its scope. This means a mutable reference can be created once the immutable +ones are no longer actively used: + +```rust +let mut s = String::from("hello"); + +let r1 = &s; +let r2 = &s; +println!("{r1}, {r2}"); // r1 and r2 last used here — they're effectively dropped + +let r3 = &mut s; // ✅ now safe, no active immutable refs +r3.push_str(", world"); +``` + +This feature is called +Non-Lexical Lifetimes (NLL) and was +introduced to make Rust's borrow checker less restrictive while keeping it safe. + +## References: + +[Why Does Rust Enforce the “One Mutable or Many Immutable References” Rule in Single-Threaded Programs?](https://users.rust-lang.org/t/why-does-rust-enforce-the-one-mutable-or-many-immutable-references-rule-in-single-threaded-programs/121017/2) + +[The Problem With Single-threaded Shared Mutability](https://manishearth.github.io/blog/2015/05/17/the-problem-with-shared-mutability/) diff --git a/memory_safety/box.md b/memory_management/box.txt diff --git a/memory_safety/drop.md b/memory_management/drop.txt diff --git a/memory_safety/interior_mutability.md b/memory_management/interior_mutability.txt diff --git a/memory_safety/iterator_invalidation.md b/memory_management/iterator_invalidation.txt diff --git a/memory_safety/mutex.md b/memory_management/mutex.txt diff --git a/memory_safety/rwlock_pattern.md b/memory_management/rwlock_pattern.txt diff --git a/memory_safety/segfault.md b/memory_management/segfault.txt diff --git a/memory_safety/shared_and_exclusive_reference.md b/memory_management/shared_and_exclusive_reference.txt diff --git a/memory_safety/spinlock.md b/memory_management/spinlock.txt diff --git a/memory_management/stack-and-heap.txt b/memory_management/stack-and-heap.txt @@ -0,0 +1,31 @@ +# Stack and Heap + +# Stack + +## Call Stack + +A region of memory the CPU uses automatically to track function calls, memory +pointers, local variables, and return addresses + +# Heap + +The heap is a region of a computer's memory used for dynamic memory allocation — +memory that is reserved and released at runtime rather than at compile time. It +is one of the two main memory areas programs use (alongside the stack), and it +gives programmers flexible control over how much memory to use and for how long. + +## Heap vs. Stack + +| Feature | Heap | Stack | +| :---------------- | :--------------------------- | :-------------------- | +| Allocation timing | Runtime (dynamic) | Compile time (static) | +| Size | Large, flexible | Small, fixed | +| Management | Manual or GC | Automatic (LIFO) | +| Access scope | Global (anywhere in program) | Local to function | +| Speed | Slightly slower | Faster | +| Risk | Memory leaks, fragmentation | Stack overflow | + +The stack follows a strict last-in, first-out (LIFO) structure, making it fast +but limited. The heap is more flexible but requires careful management — failure +to deallocate memory causes **memory leaks**, where memory becomes permanently +unavailable. diff --git a/memory_safety/string_vs_str.md b/memory_management/string_vs_str.txt diff --git a/memory_safety/vec.md b/memory_management/vec.txt diff --git a/memory_management/vec_vs_array.txt b/memory_management/vec_vs_array.txt @@ -0,0 +1,40 @@ +=============================================================================== +ARRAY vs VEC +=============================================================================== + ++-----------------+-------------------------------+-------------------------------+ +| Feature | Array `[T; N]` | Vector `Vec<T>` | ++-----------------+-------------------------------+-------------------------------+ +| Size | Fixed at compile time | Dynamic at runtime | +| Memory Location | Usually stack | Heap (pointer on stack) | +| Flexibility | Static (cannot push/pop) | Dynamic (can grow/shrink) | +| Metadata Size | 0 bytes (known at compile | 24 bytes (pointer, length, | +| | time) | capacity) | ++-----------------+-------------------------------+-------------------------------+ + +An array in Rust is a fixed-size collection of elements that all share the same +data type. The size of an array must be known at compile time, meaning it cannot +grow or shrink during program execution. + +Arrays vs Vectors in Rust + +While both arrays ([T; N]) and vectors (Vec<T>) store elements of the same type +in contiguous memory, their primary differences lie in memory management and +flexibility. Arrays are typically stored entirely on the stack, which makes them +incredibly fast to allocate but inflexible. In contrast, a vector stores its +actual elements on the heap, while keeping a small control structure on the +stack consisting of a pointer to the heap data, the current length, and the +total capacity. This allows vectors to dynamically resize at runtime using +methods like .push(), which arrays cannot do. + +Performance Implications + +Because arrays live on the stack and have no metadata, accessing their elements +is direct and avoids the pointer indirection required by a vector's heap +allocation. However, this stack allocation means that when you move an array, +you must copy every single element within it. Moving a vector is generally much +cheaper because it only requires copying the metadata pointer, length, and +capacity (usually 24 bytes), rather than the potentially massive underlying +data. Therefore, arrays are highly efficient for small, fixed-size data sets, +whereas vectors are better suited for large collections or data that changes +size over time. +\ No newline at end of file diff --git a/memory_safety/vec_vs_hashset.md b/memory_management/vec_vs_hashset.txt diff --git a/memory_safety/borrowing_rules.md b/memory_safety/borrowing_rules.md @@ -1,72 +0,0 @@ -# Borrowing Rules - -Rust enforces two hard rules at compile time: - -1. You can have any number of immutable references (&T) at the same time - -2. You can have only one mutable reference (&mut T) at a time — and when a - mutable reference is active, no immutable references may exist either - -### String example: - -#### Question: Why does rust only allow one muatble reference at a time? - -A String in Rust is heap-allocated and stores a pointer, a length, and a -capacity internally. If you hold an immutable reference r1 to a String and then -mutate it via a second mutable reference r2 — say by pushing characters — the -String may reallocate its internal buffer to a new heap address. At that point, -r1 would be pointing to freed memory, which is a dangling pointer and causes -[undefined behavior](/memory_safety/segfault.md). Rust prevents this entirely at -compile time. - -```rust -let mut s = String::from("hello"); - -let r1 = &s; // ✅ immutable borrow -let r2 = &s; // ✅ another immutable borrow — fine! -let r3 = &mut s; // ❌ compile error: can't borrow mutably while r1/r2 exist -``` - -### Furthermore: - -The single mutable reference rule actually solves several classes of bugs -simultaneously: - -- Dangling pointers — mutation causes reallocation, invalidating old references - (your example) - -- Data races — two threads mutating the same memory simultaneously leads to - unpredictable results - -- [Iterator invalidation](iterator_invalidation.md) — modifying a collection - while iterating over it (a common bug in C++ and Java) - -- Compiler optimisation safety — the compiler can safely optimise and even - vectorise (SIMD) code because it knows no two mutable aliases can overlap - -## Mutable Reference [Lifetime](/memory_safety/lifetime.md) Is Scoped - -The borrow checker is smart enough to track when a reference's last use is, not -just its scope. This means a mutable reference can be created once the immutable -ones are no longer actively used: - -```rust -let mut s = String::from("hello"); - -let r1 = &s; -let r2 = &s; -println!("{r1}, {r2}"); // r1 and r2 last used here — they're effectively dropped - -let r3 = &mut s; // ✅ now safe, no active immutable refs -r3.push_str(", world"); -``` - -This feature is called -[Non-Lexical Lifetimes (NLL)](/memory_safety/non_lexical_lifetimes.md) and was -introduced to make Rust's borrow checker less restrictive while keeping it safe. - -## References: - -[Why Does Rust Enforce the “One Mutable or Many Immutable References” Rule in Single-Threaded Programs?](https://users.rust-lang.org/t/why-does-rust-enforce-the-one-mutable-or-many-immutable-references-rule-in-single-threaded-programs/121017/2) - -[The Problem With Single-threaded Shared Mutability](https://manishearth.github.io/blog/2015/05/17/the-problem-with-shared-mutability/) diff --git a/memory_safety/stack-and-heap.md b/memory_safety/stack-and-heap.md @@ -1,35 +0,0 @@ -# Stack and Heap - -# Stack - -## Call Stack - -A region of memory the CPU uses automatically to track function calls, memory -pointers, local variables, and return addresses - -## Rust-specific rule - -[Borrowing Rules](../memory_safety/borrowing_rules.md) - -# Heap - -The heap is a region of a computer's memory used for dynamic memory allocation — -memory that is reserved and released at runtime rather than at compile time. It -is one of the two main memory areas programs use (alongside the stack), and it -gives programmers flexible control over how much memory to use and for how long. - -## Heap vs. Stack - -| Feature | Heap | Stack | -| :---------------- | :--------------------------- | :-------------------- | -| Allocation timing | Runtime (dynamic) | Compile time (static) | -| Size | Large, flexible | Small, fixed | -| Management | Manual or GC | Automatic (LIFO) | -| Access scope | Global (anywhere in program) | Local to function | -| Speed | Slightly slower | Faster | -| Risk | Memory leaks, fragmentation | Stack overflow | - -The stack follows a strict last-in, first-out (LIFO) structure, making it fast -but limited. The heap is more flexible but requires careful management — failure -to deallocate memory causes **memory leaks**, where memory becomes permanently -unavailable. diff --git a/networking/cache.md b/networking/cache.txt diff --git a/networking/ccna.md b/networking/ccna.txt diff --git a/networking/edge_infra.md b/networking/edge_infra.txt diff --git a/networking/ipc.md b/networking/ipc.md @@ -1,17 +0,0 @@ -# IPC - -## IPC (Inter-Process Communication) - -IPC is the broad concept — it refers to any mechanism that allows separate -processes to exchange data. It's not a specific technology but rather an -umbrella term covering many approaches: shared memory, message queues, pipes, -signals, and sockets. Both Unix sockets, [TCP](/networking/tcp.md) and -[UDP](/networking/udp.md) can be used as IPC mechanisms. - -## IPC (narrow/common usage) - -Local-only mechanisms (Unix sockets, pipes, shared memory) - -## TCP & UDP - -Network protocols that can do IPC, but are designed for cross-machine use diff --git a/networking/ipc.txt b/networking/ipc.txt @@ -0,0 +1,17 @@ +# IPC + +## IPC (Inter-Process Communication) + +IPC is the broad concept — it refers to any mechanism that allows separate +processes to exchange data. It's not a specific technology but rather an +umbrella term covering many approaches: shared memory, message queues, pipes, +signals, and sockets. Both Unix sockets, TCP and +UDP can be used as IPC mechanisms. + +## IPC (narrow/common usage) + +Local-only mechanisms (Unix sockets, pipes, shared memory) + +## TCP & UDP + +Network protocols that can do IPC, but are designed for cross-machine use diff --git a/networking/loopback_address.md b/networking/loopback_address.txt diff --git a/networking/subversion.md b/networking/subversion.txt diff --git a/networking/tcp.md b/networking/tcp.txt diff --git a/networking/udp.md b/networking/udp.txt diff --git a/networking/unix_socket.md b/networking/unix_socket.txt diff --git a/observability/span.md b/observability/span.txt diff --git a/observability/tracing-logs-metrics.md b/observability/tracing-logs-metrics.txt diff --git a/operating_systems/data_streams.md b/operating_systems/data_streams.txt diff --git a/operating_systems/postgres/connection_pool.md b/operating_systems/postgres/connection_pool.md @@ -1,44 +0,0 @@ -# Connection Pool - -A connection pool is a component (in your app or as a proxy like -PgBouncer/Pgpool-II) that keeps a limited number of database connections open -and reuses them across many client requests. - -## Key ideas: - -- Opening a [PostgreSQL connection](/operating_systems/postgres/postgresql_connection.md) is - relatively expensive and each connection consumes memory; creating hundreds or - thousands on demand is wasteful. - -- The pool maintains, say, 20–50 actual DB connections and lets many more - logical clients "borrow" them for short periods. - -- When a request finishes, the connection is returned to the pool instead of - being closed, so the next request can reuse it without the startup cost. - -- The pool enforces an upper bound on concurrent PostgreSQL connections, which - indirectly bounds per-connection memory usage. - -### Question: what if the pool is full? - -#### General behavior when pool is full: - -- The pool has a max number of physical PostgreSQL connections it will open. - -- When all of them are checked out and a new request comes in, most pool - implementations put the request in a queue and wait for a connection to be - returned. If no connection becomes free before a timeout, they raise an - error/exception (often something like "pool exhausted" or "timeout waiting for - connection"). - -- This is separate from PostgreSQL's own max_connections; the pool will usually - hit its own limit first and throttle clients, which is the goal. - -### Question: What if the DB itself is at max connections? - -- PostgreSQL has a server-side max_connections limit; when that is reached, any - new physical connection attempt fails with "too many connections". - -- If your pool tries to open more physical connections beyond what the server - allows (e.g. under load or after restart), those attempts will fail and bubble - up as errors to the application (connection failure rather than "queued"). diff --git a/operating_systems/postgres/connection_pool.txt b/operating_systems/postgres/connection_pool.txt @@ -0,0 +1,44 @@ +# Connection Pool + +A connection pool is a component (in your app or as a proxy like +PgBouncer/Pgpool-II) that keeps a limited number of database connections open +and reuses them across many client requests. + +## Key ideas: + +- Opening a PostgreSQL connection is + relatively expensive and each connection consumes memory; creating hundreds or + thousands on demand is wasteful. + +- The pool maintains, say, 20–50 actual DB connections and lets many more + logical clients "borrow" them for short periods. + +- When a request finishes, the connection is returned to the pool instead of + being closed, so the next request can reuse it without the startup cost. + +- The pool enforces an upper bound on concurrent PostgreSQL connections, which + indirectly bounds per-connection memory usage. + +### Question: what if the pool is full? + +#### General behavior when pool is full: + +- The pool has a max number of physical PostgreSQL connections it will open. + +- When all of them are checked out and a new request comes in, most pool + implementations put the request in a queue and wait for a connection to be + returned. If no connection becomes free before a timeout, they raise an + error/exception (often something like "pool exhausted" or "timeout waiting for + connection"). + +- This is separate from PostgreSQL's own max_connections; the pool will usually + hit its own limit first and throttle clients, which is the goal. + +### Question: What if the DB itself is at max connections? + +- PostgreSQL has a server-side max_connections limit; when that is reached, any + new physical connection attempt fails with "too many connections". + +- If your pool tries to open more physical connections beyond what the server + allows (e.g. under load or after restart), those attempts will fail and bubble + up as errors to the application (connection failure rather than "queued"). diff --git a/operating_systems/postgres/postgresql_connection.md b/operating_systems/postgres/postgresql_connection.md @@ -1,27 +0,0 @@ -# PostgreSQL Connection - -A PostgreSQL connection is the link between a client (like an app, script, or -psql) and a PostgreSQL database server that lets them send queries and receive -results. - -## Memory and Processes - -A PostgreSQL connection uses memory because each client gets its own backend -process with its own state (caches, buffers, variables, temp data, etc.), and -that state must live somewhere in RAM. This is also why too many connections can -exhaust memory and hurt performance. - -PostgreSQL is a process‑per‑connection model (on typical -[Unix-like systems](/operating_systems/README.md) a new OS process is forked for -each client connection). - -Each backend process has its own private address space for session-local state, -which avoids accidental shared mutable state between connections. - -Backends share only explicit shared-memory regions and files (e.g., shared -buffer cache, WAL, locks), with well-defined synchronization, so there's no -"mysterious" shared state like in a multi-threaded process with global -variables. - -So, you do get isolation of most state per connection, with only controlled, -intentional shared memory for common data structures. diff --git a/operating_systems/postgres/postgresql_connection.txt b/operating_systems/postgres/postgresql_connection.txt @@ -0,0 +1,27 @@ +# PostgreSQL Connection + +A PostgreSQL connection is the link between a client (like an app, script, or +psql) and a PostgreSQL database server that lets them send queries and receive +results. + +## Memory and Processes + +A PostgreSQL connection uses memory because each client gets its own backend +process with its own state (caches, buffers, variables, temp data, etc.), and +that state must live somewhere in RAM. This is also why too many connections can +exhaust memory and hurt performance. + +PostgreSQL is a process‑per‑connection model (on typical +Unix-like systems a new OS process is forked for +each client connection). + +Each backend process has its own private address space for session-local state, +which avoids accidental shared mutable state between connections. + +Backends share only explicit shared-memory regions and files (e.g., shared +buffer cache, WAL, locks), with well-defined synchronization, so there's no +"mysterious" shared state like in a multi-threaded process with global +variables. + +So, you do get isolation of most state per connection, with only controlled, +intentional shared memory for common data structures. diff --git a/operating_systems/system_call.md b/operating_systems/system_call.txt diff --git a/operating_systems/thread.md b/operating_systems/thread.txt diff --git a/operating_systems/user_space_and_kernel_space.md b/operating_systems/user_space_and_kernel_space.txt diff --git a/personal_development/4_disciplines_of_execution.md b/personal_development/4_disciplines_of_execution.txt diff --git a/personal_development/attention_restoration_theory.md b/personal_development/attention_restoration_theory.txt diff --git a/personal_development/do_less_to_do_more.md b/personal_development/do_less_to_do_more.txt diff --git a/personal_development/egoless_programming.md b/personal_development/egoless_programming.txt diff --git a/personal_development/engineering_competency.md b/personal_development/engineering_competency.txt diff --git a/personal_development/first_principles.md b/personal_development/first_principles.txt diff --git a/personal_development/learn_how_to_learn.md b/personal_development/learn_how_to_learn.txt diff --git a/personal_development/rule_of_the_vital_few.md b/personal_development/rule_of_the_vital_few.txt diff --git a/personal_development/social_media_detox.md b/personal_development/social_media_detox.txt diff --git a/personal_development/theory_of_knowledge.md b/personal_development/theory_of_knowledge.txt diff --git a/personal_development/unconscious_thought_theory.md b/personal_development/unconscious_thought_theory.txt diff --git a/personal_development/whiteboard_effect.md b/personal_development/whiteboard_effect.txt diff --git a/personal_development/zeigarnik_effect.md b/personal_development/zeigarnik_effect.txt diff --git a/quantum_mechanics/michio_kaku.md b/quantum_mechanics/michio_kaku.txt diff --git a/quantum_mechanics/simulation_theory.md b/quantum_mechanics/simulation_theory.txt diff --git a/refactoring/refactoring.md b/refactoring/refactoring.txt diff --git a/references/bookmarks.md b/references/bookmarks.txt diff --git a/references/books.md b/references/books.txt diff --git a/references/podcast.md b/references/podcast.txt diff --git a/references/rules_of_programming.md b/references/rules_of_programming.txt diff --git a/references/videos.md b/references/videos.txt diff --git a/references/videos/useful-code-is-ugly-tsoding.md b/references/videos/useful-code-is-ugly-tsoding.txt diff --git a/system_administration/commands.md b/system_administration/commands.txt diff --git a/system_administration/linux_permissions.md b/system_administration/linux_permissions.txt diff --git a/system_design/case_study/typescript_channels.md b/system_design/case_study/typescript_channels.txt diff --git a/system_design/case_study/websocket_request_trackers.md b/system_design/case_study/websocket_request_trackers.txt diff --git a/system_design/dal_architecture_overview.md b/system_design/dal_architecture_overview.md @@ -1,339 +0,0 @@ -# DAL Architecture Overview - -# Architecture Overview: Three-Layer Pattern - -The codebase uses a three-layer architecture (similar to Clean Architecture / -Layered Architecture) to separate concerns: - -``` -┌─────────────────────────────────────────────────────┐ -│ LAYER 3: NETWORKING (axum) │ -│ └── /services/projects/networking/axum/src/api/ │ -│ projects/create.rs │ -│ │ -│ Responsibility: │ -│ - HTTP endpoint handling (axum extractors) │ -│ - Authentication/Authorization (JWT tokens) │ -│ - Request/Response serialization (JSON via Axum) │ -│ - Calls core layer │ -└─────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────┐ -│ LAYER 2: CORE (Business Logic) │ -│ └── /services/projects/core/src/api/projects/ │ -│ create.rs │ -│ │ -│ Responsibility: │ -│ - Business logic validation │ -│ - Orchestration of multiple operations │ -│ - Converts domain models to/from DAL │ -│ - No HTTP/web framework knowledge │ -└────────────────────────┬────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────┐ -│ LAYER 1: DAL (Data Access Layer) │ -│ └── /layers/dal/src/models/projects/ │ -│ ├── tx_definitions.rs │ -│ └── postgres_txs.rs │ -│ │ -│ Responsibility: │ -│ - Raw SQL queries │ -│ - Database transaction management │ -│ - Zero business logic │ -└───────────┬─────────────────────────────────────────┘ - │ - ▼ -┌───────────────────────┐ -│ POSTGRESQL │ -│ DATABASE │ -└───────────────────────┘ -``` - ---- - -## How Each File Fits In - -### 1. DAL Layer: `tx_definitions.rs` + `postgres_txs.rs` - -**tx_definitions.rs** — Defines the traits that abstract database operations: - -```rust -define_dal_transactions!( - GetProjectsByDepartmentId => get_projects_by_department_id(department_id: i32) -> Vec<Project>, - CreateProject => create_project(project: NewProject) -> Project, - DeleteProject => delete_project(project_id: i32, dept_id: i32) -> bool, - CheckUserProjectAccess => check_user_project_access(user_id: i32, project_id: i32) -> bool, - GetProjectById => get_project_by_id(project_id: i32) -> Option<Project> -); -``` - -This expands to traits like: - -```rust -pub trait CreateProject { - fn create_project(project: NewProject) -> impl Future<Output = sqlx::Result<Project>> + Send; -} -``` - -See also: [define_dal_transactions!](/compiler/define_dal_transactions_macro.md) - -**postgres_txs.rs** — Implements those traits with actual SQL: - -```rust -#[db_transaction(SqlxPostGresDescriptor, CreateProject)] -async fn create_project(project: NewProject) -> Project { - let pool = T::yield_pool(); - let query = r#" - INSERT INTO projects (department_id, name, description, created_at, updated_at) - VALUES ($1, $2, $3, NOW(), NOW()) - RETURNING id, department_id, name, description, created_at, updated_at - "#; - sqlx::query_as::<_, Project>(query) - .bind(project.department_id) - .bind(project.name) - .bind(project.description) - .fetch_one(pool) - .await -} -``` - -The `#[db_transaction(StructName, TraitName)]` macro: - -1. Generates an impl `TraitName` for `StructName<T>` where - `T: YieldPostGresPool` -2. Wraps the async function body in that implementation -3. Makes the function callable as `StructName::<PoolType>::create_project(...)` - -### 2. Core Layer: `core/src/api/projects/create.rs` - -This layer orchestrates the business logic: - -```rust -pub async fn create_project<X, S>(storage_handle: &S, new_project: NewProject) -> Result<Project, NanoServiceError> -where - X: CreateProject + ProjectBranchesCreateBranch, - S: GitDataTransfer + Debug, -{ - // 1. VALIDATION (business rule) - if new_project.department_id <= 0 { - return Err(NanoServiceError::bad_request("Invalid department ID".to_string())); - } - if new_project.name.trim().is_empty() { - return Err(NanoServiceError::bad_request("Project name cannot be empty".to_string())); - } - if new_project.description.trim().is_empty() { - return Err(NanoServiceError::bad_request("Project description cannot be empty".to_string())); - } - - // 2. Create project in database - let created_project = X::create_project(new_project).await?; - - // 3. Create git directory (side effect) - create_git_repo(storage_handle, created_project.id).await?; - - // 4. Register default branch - let new_branch = NewProjectBranch { project_id: created_project.id, branch: "main".into() }; - X::create_branch(new_branch).await.map_err(|e| NanoServiceError::unknown(e.to_string()))?; - - Ok(created_project) -} -``` - -**Key characteristics:** - -- No HTTP/Websocket knowledge — pure async functions -- Generic over database handle (`X: CreateProject`) — allows mocking for tests -- Validates business rules before touching the database -- Orchestrates multiple operations (create project + git repo + branch) - -### 3. Networking Layer: `networking/axum/src/api/projects/create.rs` - -This layer adapts the core to HTTP: - -```rust -pub async fn create_project<T, X, Y>( - token: HeaderToken<X, NoRoleCheck, T>, // Auth extraction - Json(payload): Json<NewProjectRequest>, // JSON deserialization -) -> Result<impl IntoResponse, NanoServiceError> -where - T: CreateProject + GetProjectsByDepartmentId + PingAuthSession + ProjectBranchesCreateBranch, - X: GetConfigVariable, - Y: YieldPostGresPool + Send + Sync + Clone + Debug, -{ - // 1. Extract department from JWT - let department_id = token.get_department_id()?; - - // 2. Convert request DTO to domain model - let new_project = NewProject { - department_id, - name: payload.name, - description: payload.description - }; - - // 3. Create git storage handle - let storage_handle = PostgresGitBlobHandle::<Y>::new(); - - // 4. Call core business logic - let _ = create_project_core::<T, _>(&storage_handle, new_project).await?; - - // 5. Return updated list - let projects = get_projects_by_department_id_core::<T>(department_id).await?; - Ok((StatusCode::CREATED, Json(projects))) -} -``` - -**Key characteristics:** - -- Axum extractors handle HTTP parsing -- Authentication via JWT token validation -- Converts between request types (`NewProjectRequest` → `NewProject`) -- Handles HTTP concerns (status codes, JSON serialization) - ---- - -## Complete Workflow - -``` -Client Request - │ - ▼ - ┌─────────────────────────────────────────────────────────────────┐ - │ 1. HTTP REQUEST arrives at axum endpoint │ - │ POST /api/v1/projects/create │ - │ Headers: Authorization: Bearer <jwt> │ - │ Body: { "name": "...", "description": "..." } │ - └─────────────────────────────────────────────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────────────────────┐ - │ 2. AXUM LAYER (networking/axum) │ - │ - Extracts and validates JWT token │ - │ - Deserializes JSON payload │ - │ - Converts NewProjectRequest → NewProject │ - │ - Creates PostgresGitBlobHandle │ - │ - Calls create_project_core() │ - └─────────────────────────────────────────────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────────────────────┐ - │ 3. CORE LAYER (core/api) │ - │ - Validates department_id > 0 │ - │ - Validates name is not empty │ - │ - Validates description is not empty │ - │ - Calls DAL: T::create_project() │ - │ - Calls git repo creation (storage_handle) │ - │ - Calls DAL: T::create_branch() │ - │ - Returns Project model │ - └─────────────────────────────────────────────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────────────────────┐ - │ 4. DAL LAYER (dal/models) │ - │ - tx_definitions.rs: defines CreateProject trait │ - │ - postgres_txs.rs: │ - │ #[db_transaction(Struct, Trait)] │ - │ async fn create_project() -> SQL INSERT + RETURNING │ - │ - SqlxPostGresDescriptor implements the trait │ - │ - SQL executed against PostgreSQL │ - └─────────────────────────────────────────────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────────────────────┐ - │ 5. DATABASE (PostgreSQL) │ - │ INSERT INTO projects (...) VALUES (...) │ - │ RETURNING id, department_id, name, description, ... │ - └─────────────────────────────────────────────────────────────────┘ - │ - ▼ - Return back up the stack with created Project -``` - ---- - -## Data Flow Diagram - -``` -┌─────────────┐ HTTP JSON ┌─────────────┐ NewProject ┌─────────────┐ -│ Client │ ────────────────► │ networking │ ───────────────► │ core │ -│ │ │ (axum) │ │ (create) │ -└─────────────┘ └─────────────┘ └─────────────┘ - │ - ┌───────────────────────────┼─────────────────────┐ - │ │ │ - ▼ ▼ ▼ - ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ - │DAL: CreateProject│ │GitDataTransfer │ │DAL: CreateBranch │ - │(sqlx INSERT) │ │(create git dir) │ │(sqlx INSERT) │ - └──────────────────┘ └──────────────────┘ └──────────────────┘ - │ │ │ - ▼ ▼ ▼ - ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ - │ PostgreSQL │ │ Database │ │ PostgreSQL │ - │ projects │ │ git_blobs │ │ project_branches │ - └──────────────────┘ └──────────────────┘ └──────────────────┘ -``` - ---- - -## Pros and Cons of This Approach - -### ✅ Pros - -| Benefit | Explanation | -| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -| **Separation of Concerns** | Each layer has a single responsibility. DAL knows SQL, Core knows business logic, Networking knows HTTP. | -| **Testability** | Core layer can be tested with mock DB handles (`MockDeadPostGresPool`) without any HTTP server. No network needed for unit tests. | -| **Database Abstraction** | The trait-based DAL allows swapping PostgreSQL for another database (though not currently used). | -| **Reusability** | Core layer functions can be called from HTTP, WebSocket, gRPC, CLI, or tests — not coupled to HTTP. | -| **Consistency** | All endpoints follow the same pattern — predictable codebase structure. | -| **Swappable Networking** | Axum could be swapped for Actix-web or Hyper with minimal core changes. | -| **Clear Boundaries** | Easy to identify where bugs live: HTTP issue → networking, business logic → core, SQL → DAL. | - -### ❌ Cons - -| Issue | Explanation | -| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Boilerplate Overhead** | Three files per feature with traits, macros, and adapters creates ceremony. A simple CRUD operation requires significant scaffolding. | -| **Generic Proliferation** | Every function has `3+` generic type parameters (`<T, X, Y>`) making signatures hard to read and IDE autocomplete overwhelming. | -| **Tight Coupling via Traits** | The `where X: CreateProject + GetProjectsByDepartmentId + ...` clauses require implementing many traits, creating coupling between networking and DAL layers. | -| **No Transaction Across Layers** | The `create_project` core function calls multiple DAL operations that aren't wrapped in a DB transaction. If `create_git_repo` fails, the project row was already committed. | -| **Hidden Complexity in Macros** | `#[db_transaction]` and `define_dal_transactions!` are magical — hard to debug, IDE can't "go to definition" easily. | -| **Request/Response Type Proliferation** | `NewProjectRequest` (HTTP layer) → `NewProject` (Core layer) → `NewProject` (DAL) is mostly the same struct with different names. | -| **Hard to Follow the Flow** | New developers must trace through 3 files + 2 macros to understand how a simple INSERT works. | -| **Over-engineering for Simple Ops** | For a simple `SELECT * FROM projects`, you still need the full three-layer setup. | - ---- - -## Key Files Summary - -| File | Role | Key Pattern | -| -------------------------------------------------------------- | ------------------------------------------ | --------------------------------------------- | -| `layers/dal/src/models/projects/tx_definitions.rs` | Defines trait signatures for DB operations | `define_dal_transactions!` macro | -| `layers/dal/src/models/projects/postgres_txs.rs` | Implements the trait with SQL | `#[db_transaction(Struct, Trait)]` proc macro | -| `services/projects/core/src/api/projects/create.rs` | Business logic orchestration | Validation → DAL calls → Return | -| `services/projects/networking/axum/src/api/projects/create.rs` | HTTP adapter layer | Axum extractors → Call core → HTTP response | - ---- - -## Testability Example - -The beauty of this pattern is shown in the core tests: - -```rust -// Core layer test with MOCK database — no real DB needed -#[db_transaction(MockDbHandle, CreateProject)] -async fn create_project(new_project: NewProject) -> Project { - Ok(Project { id: 1, ... }) // Mocked response -} - -let result = create_project::<MockDbHandle<MockDeadPostGrosPool>, _>( - &mock_git_handle, - new_project, -) -.await; -``` - -This lets you test business logic validation and orchestration without spinning -up a PostgreSQL instance. diff --git a/system_design/dal_architecture_overview.txt b/system_design/dal_architecture_overview.txt @@ -0,0 +1,339 @@ +# DAL Architecture Overview + +# Architecture Overview: Three-Layer Pattern + +The codebase uses a three-layer architecture (similar to Clean Architecture / +Layered Architecture) to separate concerns: + +``` +┌─────────────────────────────────────────────────────┐ +│ LAYER 3: NETWORKING (axum) │ +│ └── /services/projects/networking/axum/src/api/ │ +│ projects/create.rs │ +│ │ +│ Responsibility: │ +│ - HTTP endpoint handling (axum extractors) │ +│ - Authentication/Authorization (JWT tokens) │ +│ - Request/Response serialization (JSON via Axum) │ +│ - Calls core layer │ +└─────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────┐ +│ LAYER 2: CORE (Business Logic) │ +│ └── /services/projects/core/src/api/projects/ │ +│ create.rs │ +│ │ +│ Responsibility: │ +│ - Business logic validation │ +│ - Orchestration of multiple operations │ +│ - Converts domain models to/from DAL │ +│ - No HTTP/web framework knowledge │ +└────────────────────────┬────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────┐ +│ LAYER 1: DAL (Data Access Layer) │ +│ └── /layers/dal/src/models/projects/ │ +│ ├── tx_definitions.rs │ +│ └── postgres_txs.rs │ +│ │ +│ Responsibility: │ +│ - Raw SQL queries │ +│ - Database transaction management │ +│ - Zero business logic │ +└───────────┬─────────────────────────────────────────┘ + │ + ▼ +┌───────────────────────┐ +│ POSTGRESQL │ +│ DATABASE │ +└───────────────────────┘ +``` + +--- + +## How Each File Fits In + +### 1. DAL Layer: `tx_definitions.rs` + `postgres_txs.rs` + +**tx_definitions.rs** — Defines the traits that abstract database operations: + +```rust +define_dal_transactions!( + GetProjectsByDepartmentId => get_projects_by_department_id(department_id: i32) -> Vec<Project>, + CreateProject => create_project(project: NewProject) -> Project, + DeleteProject => delete_project(project_id: i32, dept_id: i32) -> bool, + CheckUserProjectAccess => check_user_project_access(user_id: i32, project_id: i32) -> bool, + GetProjectById => get_project_by_id(project_id: i32) -> Option<Project> +); +``` + +This expands to traits like: + +```rust +pub trait CreateProject { + fn create_project(project: NewProject) -> impl Future<Output = sqlx::Result<Project>> + Send; +} +``` + +See also: define_dal_transactions! + +**postgres_txs.rs** — Implements those traits with actual SQL: + +```rust +#[db_transaction(SqlxPostGresDescriptor, CreateProject)] +async fn create_project(project: NewProject) -> Project { + let pool = T::yield_pool(); + let query = r#" + INSERT INTO projects (department_id, name, description, created_at, updated_at) + VALUES ($1, $2, $3, NOW(), NOW()) + RETURNING id, department_id, name, description, created_at, updated_at + "#; + sqlx::query_as::<_, Project>(query) + .bind(project.department_id) + .bind(project.name) + .bind(project.description) + .fetch_one(pool) + .await +} +``` + +The `#[db_transaction(StructName, TraitName)]` macro: + +1. Generates an impl `TraitName` for `StructName<T>` where + `T: YieldPostGresPool` +2. Wraps the async function body in that implementation +3. Makes the function callable as `StructName::<PoolType>::create_project(...)` + +### 2. Core Layer: `core/src/api/projects/create.rs` + +This layer orchestrates the business logic: + +```rust +pub async fn create_project<X, S>(storage_handle: &S, new_project: NewProject) -> Result<Project, NanoServiceError> +where + X: CreateProject + ProjectBranchesCreateBranch, + S: GitDataTransfer + Debug, +{ + // 1. VALIDATION (business rule) + if new_project.department_id <= 0 { + return Err(NanoServiceError::bad_request("Invalid department ID".to_string())); + } + if new_project.name.trim().is_empty() { + return Err(NanoServiceError::bad_request("Project name cannot be empty".to_string())); + } + if new_project.description.trim().is_empty() { + return Err(NanoServiceError::bad_request("Project description cannot be empty".to_string())); + } + + // 2. Create project in database + let created_project = X::create_project(new_project).await?; + + // 3. Create git directory (side effect) + create_git_repo(storage_handle, created_project.id).await?; + + // 4. Register default branch + let new_branch = NewProjectBranch { project_id: created_project.id, branch: "main".into() }; + X::create_branch(new_branch).await.map_err(|e| NanoServiceError::unknown(e.to_string()))?; + + Ok(created_project) +} +``` + +**Key characteristics:** + +- No HTTP/Websocket knowledge — pure async functions +- Generic over database handle (`X: CreateProject`) — allows mocking for tests +- Validates business rules before touching the database +- Orchestrates multiple operations (create project + git repo + branch) + +### 3. Networking Layer: `networking/axum/src/api/projects/create.rs` + +This layer adapts the core to HTTP: + +```rust +pub async fn create_project<T, X, Y>( + token: HeaderToken<X, NoRoleCheck, T>, // Auth extraction + Json(payload): Json<NewProjectRequest>, // JSON deserialization +) -> Result<impl IntoResponse, NanoServiceError> +where + T: CreateProject + GetProjectsByDepartmentId + PingAuthSession + ProjectBranchesCreateBranch, + X: GetConfigVariable, + Y: YieldPostGresPool + Send + Sync + Clone + Debug, +{ + // 1. Extract department from JWT + let department_id = token.get_department_id()?; + + // 2. Convert request DTO to domain model + let new_project = NewProject { + department_id, + name: payload.name, + description: payload.description + }; + + // 3. Create git storage handle + let storage_handle = PostgresGitBlobHandle::<Y>::new(); + + // 4. Call core business logic + let _ = create_project_core::<T, _>(&storage_handle, new_project).await?; + + // 5. Return updated list + let projects = get_projects_by_department_id_core::<T>(department_id).await?; + Ok((StatusCode::CREATED, Json(projects))) +} +``` + +**Key characteristics:** + +- Axum extractors handle HTTP parsing +- Authentication via JWT token validation +- Converts between request types (`NewProjectRequest` → `NewProject`) +- Handles HTTP concerns (status codes, JSON serialization) + +--- + +## Complete Workflow + +``` +Client Request + │ + ▼ + ┌─────────────────────────────────────────────────────────────────┐ + │ 1. HTTP REQUEST arrives at axum endpoint │ + │ POST /api/v1/projects/create │ + │ Headers: Authorization: Bearer <jwt> │ + │ Body: { "name": "...", "description": "..." } │ + └─────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌─────────────────────────────────────────────────────────────────┐ + │ 2. AXUM LAYER (networking/axum) │ + │ - Extracts and validates JWT token │ + │ - Deserializes JSON payload │ + │ - Converts NewProjectRequest → NewProject │ + │ - Creates PostgresGitBlobHandle │ + │ - Calls create_project_core() │ + └─────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌─────────────────────────────────────────────────────────────────┐ + │ 3. CORE LAYER (core/api) │ + │ - Validates department_id > 0 │ + │ - Validates name is not empty │ + │ - Validates description is not empty │ + │ - Calls DAL: T::create_project() │ + │ - Calls git repo creation (storage_handle) │ + │ - Calls DAL: T::create_branch() │ + │ - Returns Project model │ + └─────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌─────────────────────────────────────────────────────────────────┐ + │ 4. DAL LAYER (dal/models) │ + │ - tx_definitions.rs: defines CreateProject trait │ + │ - postgres_txs.rs: │ + │ #[db_transaction(Struct, Trait)] │ + │ async fn create_project() -> SQL INSERT + RETURNING │ + │ - SqlxPostGresDescriptor implements the trait │ + │ - SQL executed against PostgreSQL │ + └─────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌─────────────────────────────────────────────────────────────────┐ + │ 5. DATABASE (PostgreSQL) │ + │ INSERT INTO projects (...) VALUES (...) │ + │ RETURNING id, department_id, name, description, ... │ + └─────────────────────────────────────────────────────────────────┘ + │ + ▼ + Return back up the stack with created Project +``` + +--- + +## Data Flow Diagram + +``` +┌─────────────┐ HTTP JSON ┌─────────────┐ NewProject ┌─────────────┐ +│ Client │ ────────────────► │ networking │ ───────────────► │ core │ +│ │ │ (axum) │ │ (create) │ +└─────────────┘ └─────────────┘ └─────────────┘ + │ + ┌───────────────────────────┼─────────────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ + │DAL: CreateProject│ │GitDataTransfer │ │DAL: CreateBranch │ + │(sqlx INSERT) │ │(create git dir) │ │(sqlx INSERT) │ + └──────────────────┘ └──────────────────┘ └──────────────────┘ + │ │ │ + ▼ ▼ ▼ + ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ + │ PostgreSQL │ │ Database │ │ PostgreSQL │ + │ projects │ │ git_blobs │ │ project_branches │ + └──────────────────┘ └──────────────────┘ └──────────────────┘ +``` + +--- + +## Pros and Cons of This Approach + +### ✅ Pros + +| Benefit | Explanation | +| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| **Separation of Concerns** | Each layer has a single responsibility. DAL knows SQL, Core knows business logic, Networking knows HTTP. | +| **Testability** | Core layer can be tested with mock DB handles (`MockDeadPostGresPool`) without any HTTP server. No network needed for unit tests. | +| **Database Abstraction** | The trait-based DAL allows swapping PostgreSQL for another database (though not currently used). | +| **Reusability** | Core layer functions can be called from HTTP, WebSocket, gRPC, CLI, or tests — not coupled to HTTP. | +| **Consistency** | All endpoints follow the same pattern — predictable codebase structure. | +| **Swappable Networking** | Axum could be swapped for Actix-web or Hyper with minimal core changes. | +| **Clear Boundaries** | Easy to identify where bugs live: HTTP issue → networking, business logic → core, SQL → DAL. | + +### ❌ Cons + +| Issue | Explanation | +| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Boilerplate Overhead** | Three files per feature with traits, macros, and adapters creates ceremony. A simple CRUD operation requires significant scaffolding. | +| **Generic Proliferation** | Every function has `3+` generic type parameters (`<T, X, Y>`) making signatures hard to read and IDE autocomplete overwhelming. | +| **Tight Coupling via Traits** | The `where X: CreateProject + GetProjectsByDepartmentId + ...` clauses require implementing many traits, creating coupling between networking and DAL layers. | +| **No Transaction Across Layers** | The `create_project` core function calls multiple DAL operations that aren't wrapped in a DB transaction. If `create_git_repo` fails, the project row was already committed. | +| **Hidden Complexity in Macros** | `#[db_transaction]` and `define_dal_transactions!` are magical — hard to debug, IDE can't "go to definition" easily. | +| **Request/Response Type Proliferation** | `NewProjectRequest` (HTTP layer) → `NewProject` (Core layer) → `NewProject` (DAL) is mostly the same struct with different names. | +| **Hard to Follow the Flow** | New developers must trace through 3 files + 2 macros to understand how a simple INSERT works. | +| **Over-engineering for Simple Ops** | For a simple `SELECT * FROM projects`, you still need the full three-layer setup. | + +--- + +## Key Files Summary + +| File | Role | Key Pattern | +| -------------------------------------------------------------- | ------------------------------------------ | --------------------------------------------- | +| `layers/dal/src/models/projects/tx_definitions.rs` | Defines trait signatures for DB operations | `define_dal_transactions!` macro | +| `layers/dal/src/models/projects/postgres_txs.rs` | Implements the trait with SQL | `#[db_transaction(Struct, Trait)]` proc macro | +| `services/projects/core/src/api/projects/create.rs` | Business logic orchestration | Validation → DAL calls → Return | +| `services/projects/networking/axum/src/api/projects/create.rs` | HTTP adapter layer | Axum extractors → Call core → HTTP response | + +--- + +## Testability Example + +The beauty of this pattern is shown in the core tests: + +```rust +// Core layer test with MOCK database — no real DB needed +#[db_transaction(MockDbHandle, CreateProject)] +async fn create_project(new_project: NewProject) -> Project { + Ok(Project { id: 1, ... }) // Mocked response +} + +let result = create_project::<MockDbHandle<MockDeadPostGrosPool>, _>( + &mock_git_handle, + new_project, +) +.await; +``` + +This lets you test business logic validation and orchestration without spinning +up a PostgreSQL instance. diff --git a/system_design/design_patterns/abstraction.md b/system_design/design_patterns/abstraction.txt diff --git a/system_design/design_patterns/chain_of_responsibility.md b/system_design/design_patterns/chain_of_responsibility.txt diff --git a/system_design/design_patterns/decomposition.md b/system_design/design_patterns/decomposition.txt diff --git a/system_design/design_patterns/dependency_injection.md b/system_design/design_patterns/dependency_injection.txt diff --git a/system_design/design_patterns/design_principles.md b/system_design/design_patterns/design_principles.txt diff --git a/system_design/design_patterns/entity_boundary_interactor.md b/system_design/design_patterns/entity_boundary_interactor.txt diff --git a/system_design/design_patterns/entity_component_system.md b/system_design/design_patterns/entity_component_system.txt diff --git a/system_design/design_patterns/finite_state_machine.md b/system_design/design_patterns/finite_state_machine.txt diff --git a/system_design/design_patterns/iife.md b/system_design/design_patterns/iife.txt diff --git a/system_design/design_patterns/interface.md b/system_design/design_patterns/interface.txt diff --git a/system_design/design_patterns/mealy_and_moore_machine.md b/system_design/design_patterns/mealy_and_moore_machine.md @@ -1,202 +0,0 @@ -# Mealy and Moore Machine - -# Compare Mealy and Moore state machines with Rust code - -The key difference between Mealy and Moore -[state machines](/system_design/design_patterns/finite_state_machine.md) comes down to **where -the output lives**: in Moore machines, output is tied to the _state_; in Mealy -machines, output is tied to the _transition_ (state + input together).[^1] - ---- - -## Moore Machine - -In a **Moore machine**, each state has a fixed output associated with it, -regardless of how you arrived there or what the current input is. The output -only changes when you _enter a new state_ — making it synchronous and immune to -input glitches.[^2][^3] - -Here's a traffic light modelled as a Moore machine — the output (light colour) -depends purely on the current state: - -```rust -#[derive(Debug, Clone, PartialEq)] -enum TrafficState { - Red, - Green, - Yellow, -} - -#[derive(Debug)] -enum Input { - Timer, -} - -struct MooreMachine { - state: TrafficState, -} - -impl MooreMachine { - fn new() -> Self { - MooreMachine { state: TrafficState::Red } - } - - // Output is derived from STATE alone — no input needed - fn output(&self) -> &str { - match self.state { - TrafficState::Red => "STOP", - TrafficState::Green => "GO", - TrafficState::Yellow => "SLOW DOWN", - } - } - - // Transition: input triggers state change, output follows state - fn transition(&mut self, input: Input) { - self.state = match (&self.state, input) { - (TrafficState::Red, Input::Timer) => TrafficState::Green, - (TrafficState::Green, Input::Timer) => TrafficState::Yellow, - (TrafficState::Yellow, Input::Timer) => TrafficState::Red, - }; - // Output is read AFTER the state changes - println!("State: {:?} → Output: {}", self.state, self.output()); - } -} - -fn main() { - let mut fsm = MooreMachine::new(); - println!("Initial output: {}", fsm.output()); // "STOP" - fsm.transition(Input::Timer); // Green → "GO" - fsm.transition(Input::Timer); // Yellow → "SLOW DOWN" - fsm.transition(Input::Timer); // Red → "STOP" -} -``` - -Output is produced **after** entering the new state — `output()` takes `&self` -with no input parameter.[^3] - ---- - -## Mealy Machine - -In a **Mealy machine**, output is produced **on the transition** — it depends on -both the current state _and_ the input that triggered the move. This means -outputs can react instantly to inputs, and you typically need fewer states than -an equivalent Moore machine.[^4][^1] - -Here's a coin-operated vending machine where the output depends on both state -and the coin inserted: - -```rust -#[derive(Debug, Clone, PartialEq)] -enum VendingState { - Idle, - Has10p, - Has20p, -} - -#[derive(Debug)] -enum Coin { - P10, - P20, -} - -// Output is produced ON the transition, not from the state alone -#[derive(Debug)] -enum Output { - None, - Dispense(&'static str), - ReturnChange(u32), -} - -struct MealyMachine { - state: VendingState, -} - -impl MealyMachine { - fn new() -> Self { - MealyMachine { state: VendingState::Idle } - } - - // Returns (new_state, output) — output depends on BOTH state AND input - fn transition(&mut self, coin: Coin) -> Output { - let (next_state, output) = match (&self.state, coin) { - (VendingState::Idle, Coin::P10) => (VendingState::Has10p, Output::None), - (VendingState::Idle, Coin::P20) => (VendingState::Has20p, Output::None), - (VendingState::Has10p, Coin::P10) => (VendingState::Has20p, Output::None), - (VendingState::Has10p, Coin::P20) => (VendingState::Idle, Output::Dispense("Chewing gum (30p)")), - (VendingState::Has20p, Coin::P10) => (VendingState::Idle, Output::Dispense("Chocolate (30p)")), - (VendingState::Has20p, Coin::P20) => (VendingState::Idle, Output::ReturnChange(10)), - }; - self.state = next_state; - output - } -} - -fn main() { - let mut fsm = MealyMachine::new(); - println!("{:?}", fsm.transition(Coin::P10)); // None - println!("{:?}", fsm.transition(Coin::P20)); // Dispense("Chewing gum (30p)") - println!("{:?}", fsm.transition(Coin::P20)); // None - println!("{:?}", fsm.transition(Coin::P10)); // Dispense("Chocolate (30p)") -} -``` - -Notice `transition()` takes **both** `&self` and `coin` to compute the output — -a dead giveaway of the Mealy model.[^5] - ---- - -## Side-by-Side Comparison - -| Feature | Moore | Mealy | -| :--------------------------- | :--------------------------------------- | :------------------------------------------ | -| **Output depends on** | Current state only | Current state + input | -| **Output location** | Attached to state | Attached to transition | -| **Rust signature** | `fn output(&self)` | `fn transition(&mut self, input) -> Output` | -| **Number of states** | More (one per output combo) | Fewer [^4] | -| **Output timing** | After state change (synchronous) | Immediately on input (asynchronous) [^2] | -| **Input glitch sensitivity** | Immune | More sensitive [^2] | -| **Best for** | Display/status outputs, digital circuits | Reactive systems, protocol parsers | - ---- - -## Key Takeaway - -Both models are equally expressive — any Mealy machine can be converted to a -Moore machine by splitting states, and vice versa. In Rust, the distinction maps -cleanly: Moore output lives in a method that reads `&self` alone, while Mealy -output is returned from the transition method that takes both `&self` and the -input event.[^6][^5] -<span style="display:none">[^10][^11][^12][^13][^14][^15][^7][^8][^9]</span> - -<div align="center">⁂</div> - -[^1]: https://www.geeksforgeeks.org/theory-of-computation/difference-between-mealy-machine-and-moore-machine/ - -[^2]: https://www.youtube.com/watch?v=YiQxeuB56i0 - -[^3]: https://stackoverflow.com/questions/4009283/mealy-v-s-moore - -[^4]: https://mil.ufl.edu/3701/classes/joel/16 Lecture.pdf - -[^5]: https://comp.lang.forth.narkive.com/zJDmPu3N/mealy-vs-moore-fsm - -[^6]: https://www.reddit.com/r/explainlikeimfive/comments/30uq6e/eli5_the_difference_between_a_mealey_machine_and/ - -[^7]: https://www.youtube.com/watch?v=kb-Ww8HaHuE - -[^8]: https://forum.allaboutcircuits.com/threads/when-to-use-mealy-machine-and-when-to-use-moore-machine.190031/ - -[^9]: https://github.com/rust-cy/generic-state-machine-rs - -[^10]: https://docs.rs/rust-fsm/ - -[^11]: https://users.rust-lang.org/t/how-to-create-complex-state-machines/82714 - -[^12]: https://www.reddit.com/r/FPGA/comments/vmlb5z/is_there_any_difference_in_my_implementation_of/ - -[^13]: https://oneuptime.com/blog/post/2026-02-01-rust-state-machines/view - -[^14]: https://lib.rs/crates/edfsm - -[^15]: https://blog.devgenius.io/building-robust-distributed-state-machines-in-rust-a-comprehensive-guide-ad1a358134df diff --git a/system_design/design_patterns/mealy_and_moore_machine.txt b/system_design/design_patterns/mealy_and_moore_machine.txt @@ -0,0 +1,202 @@ +# Mealy and Moore Machine + +# Compare Mealy and Moore state machines with Rust code + +The key difference between Mealy and Moore +state machines comes down to **where +the output lives**: in Moore machines, output is tied to the _state_; in Mealy +machines, output is tied to the _transition_ (state + input together).[^1] + +--- + +## Moore Machine + +In a **Moore machine**, each state has a fixed output associated with it, +regardless of how you arrived there or what the current input is. The output +only changes when you _enter a new state_ — making it synchronous and immune to +input glitches.[^2][^3] + +Here's a traffic light modelled as a Moore machine — the output (light colour) +depends purely on the current state: + +```rust +#[derive(Debug, Clone, PartialEq)] +enum TrafficState { + Red, + Green, + Yellow, +} + +#[derive(Debug)] +enum Input { + Timer, +} + +struct MooreMachine { + state: TrafficState, +} + +impl MooreMachine { + fn new() -> Self { + MooreMachine { state: TrafficState::Red } + } + + // Output is derived from STATE alone — no input needed + fn output(&self) -> &str { + match self.state { + TrafficState::Red => "STOP", + TrafficState::Green => "GO", + TrafficState::Yellow => "SLOW DOWN", + } + } + + // Transition: input triggers state change, output follows state + fn transition(&mut self, input: Input) { + self.state = match (&self.state, input) { + (TrafficState::Red, Input::Timer) => TrafficState::Green, + (TrafficState::Green, Input::Timer) => TrafficState::Yellow, + (TrafficState::Yellow, Input::Timer) => TrafficState::Red, + }; + // Output is read AFTER the state changes + println!("State: {:?} → Output: {}", self.state, self.output()); + } +} + +fn main() { + let mut fsm = MooreMachine::new(); + println!("Initial output: {}", fsm.output()); // "STOP" + fsm.transition(Input::Timer); // Green → "GO" + fsm.transition(Input::Timer); // Yellow → "SLOW DOWN" + fsm.transition(Input::Timer); // Red → "STOP" +} +``` + +Output is produced **after** entering the new state — `output()` takes `&self` +with no input parameter.[^3] + +--- + +## Mealy Machine + +In a **Mealy machine**, output is produced **on the transition** — it depends on +both the current state _and_ the input that triggered the move. This means +outputs can react instantly to inputs, and you typically need fewer states than +an equivalent Moore machine.[^4][^1] + +Here's a coin-operated vending machine where the output depends on both state +and the coin inserted: + +```rust +#[derive(Debug, Clone, PartialEq)] +enum VendingState { + Idle, + Has10p, + Has20p, +} + +#[derive(Debug)] +enum Coin { + P10, + P20, +} + +// Output is produced ON the transition, not from the state alone +#[derive(Debug)] +enum Output { + None, + Dispense(&'static str), + ReturnChange(u32), +} + +struct MealyMachine { + state: VendingState, +} + +impl MealyMachine { + fn new() -> Self { + MealyMachine { state: VendingState::Idle } + } + + // Returns (new_state, output) — output depends on BOTH state AND input + fn transition(&mut self, coin: Coin) -> Output { + let (next_state, output) = match (&self.state, coin) { + (VendingState::Idle, Coin::P10) => (VendingState::Has10p, Output::None), + (VendingState::Idle, Coin::P20) => (VendingState::Has20p, Output::None), + (VendingState::Has10p, Coin::P10) => (VendingState::Has20p, Output::None), + (VendingState::Has10p, Coin::P20) => (VendingState::Idle, Output::Dispense("Chewing gum (30p)")), + (VendingState::Has20p, Coin::P10) => (VendingState::Idle, Output::Dispense("Chocolate (30p)")), + (VendingState::Has20p, Coin::P20) => (VendingState::Idle, Output::ReturnChange(10)), + }; + self.state = next_state; + output + } +} + +fn main() { + let mut fsm = MealyMachine::new(); + println!("{:?}", fsm.transition(Coin::P10)); // None + println!("{:?}", fsm.transition(Coin::P20)); // Dispense("Chewing gum (30p)") + println!("{:?}", fsm.transition(Coin::P20)); // None + println!("{:?}", fsm.transition(Coin::P10)); // Dispense("Chocolate (30p)") +} +``` + +Notice `transition()` takes **both** `&self` and `coin` to compute the output — +a dead giveaway of the Mealy model.[^5] + +--- + +## Side-by-Side Comparison + +| Feature | Moore | Mealy | +| :--------------------------- | :--------------------------------------- | :------------------------------------------ | +| **Output depends on** | Current state only | Current state + input | +| **Output location** | Attached to state | Attached to transition | +| **Rust signature** | `fn output(&self)` | `fn transition(&mut self, input) -> Output` | +| **Number of states** | More (one per output combo) | Fewer [^4] | +| **Output timing** | After state change (synchronous) | Immediately on input (asynchronous) [^2] | +| **Input glitch sensitivity** | Immune | More sensitive [^2] | +| **Best for** | Display/status outputs, digital circuits | Reactive systems, protocol parsers | + +--- + +## Key Takeaway + +Both models are equally expressive — any Mealy machine can be converted to a +Moore machine by splitting states, and vice versa. In Rust, the distinction maps +cleanly: Moore output lives in a method that reads `&self` alone, while Mealy +output is returned from the transition method that takes both `&self` and the +input event.[^6][^5] +<span style="display:none">[^10][^11][^12][^13][^14][^15][^7][^8][^9]</span> + +<div align="center">⁂</div> + +[^1]: https://www.geeksforgeeks.org/theory-of-computation/difference-between-mealy-machine-and-moore-machine/ + +[^2]: https://www.youtube.com/watch?v=YiQxeuB56i0 + +[^3]: https://stackoverflow.com/questions/4009283/mealy-v-s-moore + +[^4]: https://mil.ufl.edu/3701/classes/joel/16 Lecture.pdf + +[^5]: https://comp.lang.forth.narkive.com/zJDmPu3N/mealy-vs-moore-fsm + +[^6]: https://www.reddit.com/r/explainlikeimfive/comments/30uq6e/eli5_the_difference_between_a_mealey_machine_and/ + +[^7]: https://www.youtube.com/watch?v=kb-Ww8HaHuE + +[^8]: https://forum.allaboutcircuits.com/threads/when-to-use-mealy-machine-and-when-to-use-moore-machine.190031/ + +[^9]: https://github.com/rust-cy/generic-state-machine-rs + +[^10]: https://docs.rs/rust-fsm/ + +[^11]: https://users.rust-lang.org/t/how-to-create-complex-state-machines/82714 + +[^12]: https://www.reddit.com/r/FPGA/comments/vmlb5z/is_there_any_difference_in_my_implementation_of/ + +[^13]: https://oneuptime.com/blog/post/2026-02-01-rust-state-machines/view + +[^14]: https://lib.rs/crates/edfsm + +[^15]: https://blog.devgenius.io/building-robust-distributed-state-machines-in-rust-a-comprehensive-guide-ad1a358134df diff --git a/system_design/design_patterns/revealing_module_pattern.md b/system_design/design_patterns/revealing_module_pattern.txt diff --git a/system_design/edge_architecture.md b/system_design/edge_architecture.txt diff --git a/system_design/postgresql_transactions_acid.md b/system_design/postgresql_transactions_acid.txt diff --git a/system_design/server_authoritative_design.md b/system_design/server_authoritative_design.txt diff --git a/system_design/server_scaling.md b/system_design/server_scaling.md @@ -1,29 +0,0 @@ -# Server Scaling - -## Load Balancer - -A load balancer evenly distributes incoming traffic among web servers that are -defined in a load-balanced set. - -## Database Replication - -A master database generally only supports write operations. A slave database -gets copies of the data from the master database and only supports read -operations. - -<img src="/assets/system_design-server_scaling.png" alt="Server Setup" width="100%"> - -## Cache Tier - -The [cache](/networking/cache.md) tier is a temporary data store layer, much -faster than the database. The benefits of having a separate cache tier include -better system performance, ability to reduce database workloads, and the ability -to scale the cache tier independently. - -## Content Delivery Network (CDN) - -A CDN is a network of geographically dispersed servers used to deliver static -content. CDN servers cache static content like images, videos, CSS, JavaScript -files, etc. - -<img src="/assets/cdn_cache_server.png" alt="CDN Cache Server Setup" width="100%"> diff --git a/system_design/server_scaling.txt b/system_design/server_scaling.txt @@ -0,0 +1,29 @@ +# Server Scaling + +## Load Balancer + +A load balancer evenly distributes incoming traffic among web servers that are +defined in a load-balanced set. + +## Database Replication + +A master database generally only supports write operations. A slave database +gets copies of the data from the master database and only supports read +operations. + +<img src="/assets/system_design-server_scaling.png" alt="Server Setup" width="100%"> + +## Cache Tier + +The cache tier is a temporary data store layer, much +faster than the database. The benefits of having a separate cache tier include +better system performance, ability to reduce database workloads, and the ability +to scale the cache tier independently. + +## Content Delivery Network (CDN) + +A CDN is a network of geographically dispersed servers used to deliver static +content. CDN servers cache static content like images, videos, CSS, JavaScript +files, etc. + +<img src="/assets/cdn_cache_server.png" alt="CDN Cache Server Setup" width="100%"> diff --git a/system_design/single_server.md b/system_design/single_server.txt diff --git a/system_design/state_management/action_reducer_pattern.md b/system_design/state_management/action_reducer_pattern.txt diff --git a/system_design/state_management/broadcastchannel_api.md b/system_design/state_management/broadcastchannel_api.txt diff --git a/system_design/state_management/observer_pattern.md b/system_design/state_management/observer_pattern.txt diff --git a/system_design/state_management/path_based_state.md b/system_design/state_management/path_based_state.txt diff --git a/system_design/state_management/persistent_state.md b/system_design/state_management/persistent_state.txt diff --git a/system_design/state_management/proxy_based_reactivity.md b/system_design/state_management/proxy_based_reactivity.txt diff --git a/system_design/state_management/singleton_channels.md b/system_design/state_management/singleton_channels.txt diff --git a/system_design/state_management/svelte_state.md b/system_design/state_management/svelte_state.txt diff --git a/test_driven_development/arrange_act_assert.md b/test_driven_development/arrange_act_assert.txt diff --git a/test_driven_development/unit_testing.md b/test_driven_development/unit_testing.md @@ -1,71 +0,0 @@ -# Unit Testing - -## Writing Isolated and Focused Unit Tests - -Unit tests should be **isolated and focused**, testing one small, well-defined -unit of functionality at a time. Each test should verify a single behavior -without relying on other parts of the system. When tests are too broad or -tightly coupled across components, they become brittle — a minor change in one -area can cause unrelated tests to fail. - -To ensure reliability and maintainability, unit tests should: - -- Run independently of external systems or global state. -- Use **mocks or stubs** to replace dependencies. -- Follow the - [**Arrange–Act–Assert**](/test_driven_development/arrange_act_assert.md) - pattern for clarity. -- Be **deterministic** and **fast** so they can run often during development. - -Writing tests this way builds confidence in each unit, simplifies debugging, and -supports modular, testable code design. - ---- - -## Enabling CI/CD Test Environments with Isolated Resources - -For continuous integration workflows such as **GitHub Actions** or **Forgejo**, -unit tests should run within a fully isolated and reproducible environment. This -means the application should be capable of **spinning up a dedicated test -PostgreSQL instance** and a **test server API** that operates on a separate port -with its own **test-specific environment variables**. - -These configurations ensure that test runs do not interfere with production or -staging databases. The test infrastructure should start up quickly and shut down -cleanly as part of the CI/CD pipeline, allowing automated workflows to execute -the full test suite independently for every build or pull request. This approach -guarantees repeatable, safe testing while maintaining complete separation -between test and production systems. - ---- - -### Question: What's the difference between intergation tests and unit tests? - -Unit tests focus on small, isolated pieces of code and run very fast, so they -give precise, quick feedback and make it easy to iterate or refactor without -breaking unrelated behavior. In contrast, integration tests exercise multiple -components together (like API, database, and services) to verify real workflows, -which provides higher confidence that the system behaves correctly as a whole. - -The tradeoff is that integration tests are slower, more complex to set up, and -failures can be harder to diagnose, since a small change in one part of the -system can break a test somewhere else. Because of this, you typically rely on -unit tests for rapid development of specific functionality, and use integration -tests more sparingly to ensure that the integrated application still works end -to end. - ---- - -## Abstractions and Patterns to Keep Tests Stable - -This is where **interfaces** (traits), the facade and adapter patterns, -dependency injection, and internal mutability come in. When a piece of code is -wrapped behind a stable trait or adapter interface, the call sites that depend -on it can remain unchanged, while other developers are free to change the -internals of the wrapped functions without breaking those callers. This gives -more confidence in integration testing when integrating with different external -systems, call sites, or frontends, because the surface contracts stay stable -even as implementations evolve. These abstraction layers reinforce separation of -concerns between core functions, business logic, infrastructure, and UI/API -layers, so each layer can be unit-tested in isolation and then composed -predictably in integration tests. diff --git a/test_driven_development/unit_testing.txt b/test_driven_development/unit_testing.txt @@ -0,0 +1,71 @@ +# Unit Testing + +## Writing Isolated and Focused Unit Tests + +Unit tests should be **isolated and focused**, testing one small, well-defined +unit of functionality at a time. Each test should verify a single behavior +without relying on other parts of the system. When tests are too broad or +tightly coupled across components, they become brittle — a minor change in one +area can cause unrelated tests to fail. + +To ensure reliability and maintainability, unit tests should: + +- Run independently of external systems or global state. +- Use **mocks or stubs** to replace dependencies. +- Follow the + **Arrange–Act–Assert** + pattern for clarity. +- Be **deterministic** and **fast** so they can run often during development. + +Writing tests this way builds confidence in each unit, simplifies debugging, and +supports modular, testable code design. + +--- + +## Enabling CI/CD Test Environments with Isolated Resources + +For continuous integration workflows such as **GitHub Actions** or **Forgejo**, +unit tests should run within a fully isolated and reproducible environment. This +means the application should be capable of **spinning up a dedicated test +PostgreSQL instance** and a **test server API** that operates on a separate port +with its own **test-specific environment variables**. + +These configurations ensure that test runs do not interfere with production or +staging databases. The test infrastructure should start up quickly and shut down +cleanly as part of the CI/CD pipeline, allowing automated workflows to execute +the full test suite independently for every build or pull request. This approach +guarantees repeatable, safe testing while maintaining complete separation +between test and production systems. + +--- + +### Question: What's the difference between intergation tests and unit tests? + +Unit tests focus on small, isolated pieces of code and run very fast, so they +give precise, quick feedback and make it easy to iterate or refactor without +breaking unrelated behavior. In contrast, integration tests exercise multiple +components together (like API, database, and services) to verify real workflows, +which provides higher confidence that the system behaves correctly as a whole. + +The tradeoff is that integration tests are slower, more complex to set up, and +failures can be harder to diagnose, since a small change in one part of the +system can break a test somewhere else. Because of this, you typically rely on +unit tests for rapid development of specific functionality, and use integration +tests more sparingly to ensure that the integrated application still works end +to end. + +--- + +## Abstractions and Patterns to Keep Tests Stable + +This is where **interfaces** (traits), the facade and adapter patterns, +dependency injection, and internal mutability come in. When a piece of code is +wrapped behind a stable trait or adapter interface, the call sites that depend +on it can remain unchanged, while other developers are free to change the +internals of the wrapped functions without breaking those callers. This gives +more confidence in integration testing when integrating with different external +systems, call sites, or frontends, because the surface contracts stay stable +even as implementations evolve. These abstraction layers reinforce separation of +concerns between core functions, business logic, infrastructure, and UI/API +layers, so each layer can be unit-tested in isolation and then composed +predictably in integration tests. diff --git a/web_development/http_headers.md b/web_development/http_headers.txt diff --git a/web_development/openapi-guide.md b/web_development/openapi-guide.txt diff --git a/web_development/svelte.md b/web_development/svelte.txt diff --git a/web_development/svelte_state.md b/web_development/svelte_state.txt diff --git a/web_development/typescript.md b/web_development/typescript.txt diff --git a/web_development/typescript_books.md b/web_development/typescript_books.txt diff --git a/z-md/agent.md b/z-md/agent.txt