dal_architecture_overview.md (20119B)
1 # DAL Architecture Overview 2 3 # Architecture Overview: Three-Layer Pattern 4 5 The codebase uses a three-layer architecture (similar to Clean Architecture / 6 Layered Architecture) to separate concerns: 7 8 ``` 9 ┌─────────────────────────────────────────────────────┐ 10 │ LAYER 3: NETWORKING (axum) │ 11 │ └── /services/projects/networking/axum/src/api/ │ 12 │ projects/create.rs │ 13 │ │ 14 │ Responsibility: │ 15 │ - HTTP endpoint handling (axum extractors) │ 16 │ - Authentication/Authorization (JWT tokens) │ 17 │ - Request/Response serialization (JSON via Axum) │ 18 │ - Calls core layer │ 19 └─────────────────────────────────────────────────────┘ 20 │ 21 ▼ 22 ┌─────────────────────────────────────────────────────┐ 23 │ LAYER 2: CORE (Business Logic) │ 24 │ └── /services/projects/core/src/api/projects/ │ 25 │ create.rs │ 26 │ │ 27 │ Responsibility: │ 28 │ - Business logic validation │ 29 │ - Orchestration of multiple operations │ 30 │ - Converts domain models to/from DAL │ 31 │ - No HTTP/web framework knowledge │ 32 └────────────────────────┬────────────────────────────┘ 33 │ 34 ▼ 35 ┌─────────────────────────────────────────────────────┐ 36 │ LAYER 1: DAL (Data Access Layer) │ 37 │ └── /layers/dal/src/models/projects/ │ 38 │ ├── tx_definitions.rs │ 39 │ └── postgres_txs.rs │ 40 │ │ 41 │ Responsibility: │ 42 │ - Raw SQL queries │ 43 │ - Database transaction management │ 44 │ - Zero business logic │ 45 └───────────┬─────────────────────────────────────────┘ 46 │ 47 ▼ 48 ┌───────────────────────┐ 49 │ POSTGRESQL │ 50 │ DATABASE │ 51 └───────────────────────┘ 52 ``` 53 54 --- 55 56 ## How Each File Fits In 57 58 ### 1. DAL Layer: `tx_definitions.rs` + `postgres_txs.rs` 59 60 **tx_definitions.rs** — Defines the traits that abstract database operations: 61 62 ```rust 63 define_dal_transactions!( 64 GetProjectsByDepartmentId => get_projects_by_department_id(department_id: i32) -> Vec<Project>, 65 CreateProject => create_project(project: NewProject) -> Project, 66 DeleteProject => delete_project(project_id: i32, dept_id: i32) -> bool, 67 CheckUserProjectAccess => check_user_project_access(user_id: i32, project_id: i32) -> bool, 68 GetProjectById => get_project_by_id(project_id: i32) -> Option<Project> 69 ); 70 ``` 71 72 This expands to traits like: 73 74 ```rust 75 pub trait CreateProject { 76 fn create_project(project: NewProject) -> impl Future<Output = sqlx::Result<Project>> + Send; 77 } 78 ``` 79 80 See also: [define_dal_transactions!](/compiler/define_dal_transactions_macro.md) 81 82 **postgres_txs.rs** — Implements those traits with actual SQL: 83 84 ```rust 85 #[db_transaction(SqlxPostGresDescriptor, CreateProject)] 86 async fn create_project(project: NewProject) -> Project { 87 let pool = T::yield_pool(); 88 let query = r#" 89 INSERT INTO projects (department_id, name, description, created_at, updated_at) 90 VALUES ($1, $2, $3, NOW(), NOW()) 91 RETURNING id, department_id, name, description, created_at, updated_at 92 "#; 93 sqlx::query_as::<_, Project>(query) 94 .bind(project.department_id) 95 .bind(project.name) 96 .bind(project.description) 97 .fetch_one(pool) 98 .await 99 } 100 ``` 101 102 The `#[db_transaction(StructName, TraitName)]` macro: 103 104 1. Generates an impl `TraitName` for `StructName<T>` where 105 `T: YieldPostGresPool` 106 2. Wraps the async function body in that implementation 107 3. Makes the function callable as `StructName::<PoolType>::create_project(...)` 108 109 ### 2. Core Layer: `core/src/api/projects/create.rs` 110 111 This layer orchestrates the business logic: 112 113 ```rust 114 pub async fn create_project<X, S>(storage_handle: &S, new_project: NewProject) -> Result<Project, NanoServiceError> 115 where 116 X: CreateProject + ProjectBranchesCreateBranch, 117 S: GitDataTransfer + Debug, 118 { 119 // 1. VALIDATION (business rule) 120 if new_project.department_id <= 0 { 121 return Err(NanoServiceError::bad_request("Invalid department ID".to_string())); 122 } 123 if new_project.name.trim().is_empty() { 124 return Err(NanoServiceError::bad_request("Project name cannot be empty".to_string())); 125 } 126 if new_project.description.trim().is_empty() { 127 return Err(NanoServiceError::bad_request("Project description cannot be empty".to_string())); 128 } 129 130 // 2. Create project in database 131 let created_project = X::create_project(new_project).await?; 132 133 // 3. Create git directory (side effect) 134 create_git_repo(storage_handle, created_project.id).await?; 135 136 // 4. Register default branch 137 let new_branch = NewProjectBranch { project_id: created_project.id, branch: "main".into() }; 138 X::create_branch(new_branch).await.map_err(|e| NanoServiceError::unknown(e.to_string()))?; 139 140 Ok(created_project) 141 } 142 ``` 143 144 **Key characteristics:** 145 146 - No HTTP/Websocket knowledge — pure async functions 147 - Generic over database handle (`X: CreateProject`) — allows mocking for tests 148 - Validates business rules before touching the database 149 - Orchestrates multiple operations (create project + git repo + branch) 150 151 ### 3. Networking Layer: `networking/axum/src/api/projects/create.rs` 152 153 This layer adapts the core to HTTP: 154 155 ```rust 156 pub async fn create_project<T, X, Y>( 157 token: HeaderToken<X, NoRoleCheck, T>, // Auth extraction 158 Json(payload): Json<NewProjectRequest>, // JSON deserialization 159 ) -> Result<impl IntoResponse, NanoServiceError> 160 where 161 T: CreateProject + GetProjectsByDepartmentId + PingAuthSession + ProjectBranchesCreateBranch, 162 X: GetConfigVariable, 163 Y: YieldPostGresPool + Send + Sync + Clone + Debug, 164 { 165 // 1. Extract department from JWT 166 let department_id = token.get_department_id()?; 167 168 // 2. Convert request DTO to domain model 169 let new_project = NewProject { 170 department_id, 171 name: payload.name, 172 description: payload.description 173 }; 174 175 // 3. Create git storage handle 176 let storage_handle = PostgresGitBlobHandle::<Y>::new(); 177 178 // 4. Call core business logic 179 let _ = create_project_core::<T, _>(&storage_handle, new_project).await?; 180 181 // 5. Return updated list 182 let projects = get_projects_by_department_id_core::<T>(department_id).await?; 183 Ok((StatusCode::CREATED, Json(projects))) 184 } 185 ``` 186 187 **Key characteristics:** 188 189 - Axum extractors handle HTTP parsing 190 - Authentication via JWT token validation 191 - Converts between request types (`NewProjectRequest` → `NewProject`) 192 - Handles HTTP concerns (status codes, JSON serialization) 193 194 --- 195 196 ## Complete Workflow 197 198 ``` 199 Client Request 200 │ 201 ▼ 202 ┌─────────────────────────────────────────────────────────────────┐ 203 │ 1. HTTP REQUEST arrives at axum endpoint │ 204 │ POST /api/v1/projects/create │ 205 │ Headers: Authorization: Bearer <jwt> │ 206 │ Body: { "name": "...", "description": "..." } │ 207 └─────────────────────────────────────────────────────────────────┘ 208 │ 209 ▼ 210 ┌─────────────────────────────────────────────────────────────────┐ 211 │ 2. AXUM LAYER (networking/axum) │ 212 │ - Extracts and validates JWT token │ 213 │ - Deserializes JSON payload │ 214 │ - Converts NewProjectRequest → NewProject │ 215 │ - Creates PostgresGitBlobHandle │ 216 │ - Calls create_project_core() │ 217 └─────────────────────────────────────────────────────────────────┘ 218 │ 219 ▼ 220 ┌─────────────────────────────────────────────────────────────────┐ 221 │ 3. CORE LAYER (core/api) │ 222 │ - Validates department_id > 0 │ 223 │ - Validates name is not empty │ 224 │ - Validates description is not empty │ 225 │ - Calls DAL: T::create_project() │ 226 │ - Calls git repo creation (storage_handle) │ 227 │ - Calls DAL: T::create_branch() │ 228 │ - Returns Project model │ 229 └─────────────────────────────────────────────────────────────────┘ 230 │ 231 ▼ 232 ┌─────────────────────────────────────────────────────────────────┐ 233 │ 4. DAL LAYER (dal/models) │ 234 │ - tx_definitions.rs: defines CreateProject trait │ 235 │ - postgres_txs.rs: │ 236 │ #[db_transaction(Struct, Trait)] │ 237 │ async fn create_project() -> SQL INSERT + RETURNING │ 238 │ - SqlxPostGresDescriptor implements the trait │ 239 │ - SQL executed against PostgreSQL │ 240 └─────────────────────────────────────────────────────────────────┘ 241 │ 242 ▼ 243 ┌─────────────────────────────────────────────────────────────────┐ 244 │ 5. DATABASE (PostgreSQL) │ 245 │ INSERT INTO projects (...) VALUES (...) │ 246 │ RETURNING id, department_id, name, description, ... │ 247 └─────────────────────────────────────────────────────────────────┘ 248 │ 249 ▼ 250 Return back up the stack with created Project 251 ``` 252 253 --- 254 255 ## Data Flow Diagram 256 257 ``` 258 ┌─────────────┐ HTTP JSON ┌─────────────┐ NewProject ┌─────────────┐ 259 │ Client │ ────────────────► │ networking │ ───────────────► │ core │ 260 │ │ │ (axum) │ │ (create) │ 261 └─────────────┘ └─────────────┘ └─────────────┘ 262 │ 263 ┌───────────────────────────┼─────────────────────┐ 264 │ │ │ 265 ▼ ▼ ▼ 266 ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ 267 │DAL: CreateProject│ │GitDataTransfer │ │DAL: CreateBranch │ 268 │(sqlx INSERT) │ │(create git dir) │ │(sqlx INSERT) │ 269 └──────────────────┘ └──────────────────┘ └──────────────────┘ 270 │ │ │ 271 ▼ ▼ ▼ 272 ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ 273 │ PostgreSQL │ │ Database │ │ PostgreSQL │ 274 │ projects │ │ git_blobs │ │ project_branches │ 275 └──────────────────┘ └──────────────────┘ └──────────────────┘ 276 ``` 277 278 --- 279 280 ## Pros and Cons of This Approach 281 282 ### ✅ Pros 283 284 | Benefit | Explanation | 285 | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | 286 | **Separation of Concerns** | Each layer has a single responsibility. DAL knows SQL, Core knows business logic, Networking knows HTTP. | 287 | **Testability** | Core layer can be tested with mock DB handles (`MockDeadPostGresPool`) without any HTTP server. No network needed for unit tests. | 288 | **Database Abstraction** | The trait-based DAL allows swapping PostgreSQL for another database (though not currently used). | 289 | **Reusability** | Core layer functions can be called from HTTP, WebSocket, gRPC, CLI, or tests — not coupled to HTTP. | 290 | **Consistency** | All endpoints follow the same pattern — predictable codebase structure. | 291 | **Swappable Networking** | Axum could be swapped for Actix-web or Hyper with minimal core changes. | 292 | **Clear Boundaries** | Easy to identify where bugs live: HTTP issue → networking, business logic → core, SQL → DAL. | 293 294 ### ❌ Cons 295 296 | Issue | Explanation | 297 | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | 298 | **Boilerplate Overhead** | Three files per feature with traits, macros, and adapters creates ceremony. A simple CRUD operation requires significant scaffolding. | 299 | **Generic Proliferation** | Every function has `3+` generic type parameters (`<T, X, Y>`) making signatures hard to read and IDE autocomplete overwhelming. | 300 | **Tight Coupling via Traits** | The `where X: CreateProject + GetProjectsByDepartmentId + ...` clauses require implementing many traits, creating coupling between networking and DAL layers. | 301 | **No Transaction Across Layers** | The `create_project` core function calls multiple DAL operations that aren't wrapped in a DB transaction. If `create_git_repo` fails, the project row was already committed. | 302 | **Hidden Complexity in Macros** | `#[db_transaction]` and `define_dal_transactions!` are magical — hard to debug, IDE can't "go to definition" easily. | 303 | **Request/Response Type Proliferation** | `NewProjectRequest` (HTTP layer) → `NewProject` (Core layer) → `NewProject` (DAL) is mostly the same struct with different names. | 304 | **Hard to Follow the Flow** | New developers must trace through 3 files + 2 macros to understand how a simple INSERT works. | 305 | **Over-engineering for Simple Ops** | For a simple `SELECT * FROM projects`, you still need the full three-layer setup. | 306 307 --- 308 309 ## Key Files Summary 310 311 | File | Role | Key Pattern | 312 | -------------------------------------------------------------- | ------------------------------------------ | --------------------------------------------- | 313 | `layers/dal/src/models/projects/tx_definitions.rs` | Defines trait signatures for DB operations | `define_dal_transactions!` macro | 314 | `layers/dal/src/models/projects/postgres_txs.rs` | Implements the trait with SQL | `#[db_transaction(Struct, Trait)]` proc macro | 315 | `services/projects/core/src/api/projects/create.rs` | Business logic orchestration | Validation → DAL calls → Return | 316 | `services/projects/networking/axum/src/api/projects/create.rs` | HTTP adapter layer | Axum extractors → Call core → HTTP response | 317 318 --- 319 320 ## Testability Example 321 322 The beauty of this pattern is shown in the core tests: 323 324 ```rust 325 // Core layer test with MOCK database — no real DB needed 326 #[db_transaction(MockDbHandle, CreateProject)] 327 async fn create_project(new_project: NewProject) -> Project { 328 Ok(Project { id: 1, ... }) // Mocked response 329 } 330 331 let result = create_project::<MockDbHandle<MockDeadPostGrosPool>, _>( 332 &mock_git_handle, 333 new_project, 334 ) 335 .await; 336 ``` 337 338 This lets you test business logic validation and orchestration without spinning 339 up a PostgreSQL instance.