From c57bd5229f5af6a3c0ce1ce82fe1f91b90f64070 Mon Sep 17 00:00:00 2001 From: emil28092005 Date: Fri, 19 Jun 2026 17:02:33 +0300 Subject: [PATCH] docs: update all docs for publication MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README.md: new — features, quick start, controls, MCP tools, requirements - AGENTS.md: rewritten for pure P/Invoke Vulkan 1.3 (was Raylib/OpenGL) - CORTEX_ENGINE_ARCHITECTURE.md: complete rewrite — current architecture, frame loop, UBO layout, push constants, shadow mapping, PBR, AI/MCP, physics, ImGui, video recording, content, testing - VULKAN_IMPLEMENTATION_PLAN.md: marked as COMPLETE with all 21 phases - scripts/run.sh: updated examples - .gitignore: added Videos/, imgui.ini, cortex.mp4 - Removed tracked imgui.ini and video files --- .gitignore | 7 + AGENTS.md | 54 +- CORTEX_ENGINE_ARCHITECTURE.md | 1079 +++++++-------------------------- README.md | 75 +++ VULKAN_IMPLEMENTATION_PLAN.md | 754 +---------------------- imgui.ini | 40 -- scripts/run.sh | 7 +- 7 files changed, 368 insertions(+), 1648 deletions(-) create mode 100644 README.md delete mode 100644 imgui.ini diff --git a/.gitignore b/.gitignore index 6abaf68..f06a6b7 100644 --- a/.gitignore +++ b/.gitignore @@ -18,6 +18,12 @@ Thumbs.db # Generated screenshots Screenshots/ +# Video recordings +Videos/ + +# ImGui +imgui.ini + # dotnet *.dll *.exe @@ -30,3 +36,4 @@ Screenshots/ !claude_desktop_config.json .playwright-mcp/ Vulkan-Guide/ +cortex.mp4 diff --git a/AGENTS.md b/AGENTS.md index 79dfeb9..b051ba4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ ## Project Overview -Cortex Engine is a C# (.NET 9) AI-Native 3D game engine. The primary render backend is Raylib-cs (OpenGL). A Vulkan backend exists but is deferred. +Cortex Engine is a C# (.NET 9) AI-Native 3D game engine with a pure P/Invoke Vulkan 1.3 render backend. No wrapper libraries (Silk.NET, Vortice, OpenTK) — direct Vulkan API calls via `vkGetInstanceProcAddr`/`vkGetDeviceProcAddr`. ## Build Commands @@ -10,31 +10,28 @@ Cortex Engine is a C# (.NET 9) AI-Native 3D game engine. The primary render back # Build (Debug) dotnet build CORTEX_ENGINE.sln -c Debug -# Build (Release) -dotnet build CORTEX_ENGINE.sln -c Release - # Run the engine ./scripts/run.sh -# Run with test scene + camera tour (headless screenshot capture) -dotnet run --project src/CortexEngine.App/CortexEngine.App.csproj -c Release -- --test-scene --camera-tour --mcp-port 0 +# Run with MCP server (AI control via HTTP/SSE) +./scripts/run.sh -- --mcp-port 5000 -# Run with MCP server -./scripts/run.sh --mcp-port 5000 +# Run tests +dotnet test tests/Engine.Tests/Engine.Tests.csproj -c Debug ``` ## Lint / Typecheck -No separate lint command. `dotnet build` with 0 warnings is the standard. Run `dotnet build CORTEX_ENGINE.sln -c Release` to verify. +No separate lint command. `dotnet build` with 0 errors is the standard. Run `dotnet build CORTEX_ENGINE.sln -c Debug` to verify. 227 xUnit tests cover Vulkan struct sizes, enum values, OBJ loading, vertex layout, shadow mapping, camera controllers, AI commands. ## Architecture -- **Engine.Core** — `IWindow`, `IInputState`, `Key` enum, `Sdl3Window`, camera controllers, ECS components (`Transform`, `Mesh`, `Material`, `Light`, `Camera`), `Timing` -- **Engine.Graphics** — HAL interfaces (`IRenderContext`, `IRenderer`), `RenderBackendFactory`, mesh loaders (`ObjLoader`, `GltfLoader`) -- **Engine.Graphics.Raylib** — Primary backend. `RaylibWindow` (GLFW), `RaylibInputState`, `RaylibRenderer` with custom GLSL 330 shader (Fresnel, ACES, gamma) -- **Engine.Graphics.Vulkan** — Deferred backend. Compiles but untested. Uses `Sdl3Window` for Vulkan surface. -- **Engine.AI** — `AiCommandProcessor` (7 commands), MCP HTTP + stdio servers -- **CortexEngine.App** — Entry point, main loop, scene setup +- **Engine.Core** — `IWindow`, `IInputState`, `Key` enum, `Sdl3Window` (SDL3 + Vulkan surface), camera controllers (`FreeFly`, `Orbit`), ECS components (`Transform`, `Mesh`, `Material`, `Light`, `Camera`, `RigidBody`), `Vertex` struct, `Timing` +- **Engine.Graphics** — HAL interfaces (`IRenderContext`, `IRenderer`, `IScreenshotProvider`), `RenderBackendFactory`, `ObjLoader`, `MeshMath`, `ProceduralMesh`, `SceneSerializer` +- **Engine.Graphics.Vulkan** — Pure P/Invoke Vulkan 1.3 backend. Vulkan 1.3 features: dynamic rendering, synchronization2, imageCubeArray. Multi-light PBR with cubemap array shadows. ImGui integration. Video recording via FFmpeg pipe. +- **Engine.Physics** — JoltPhysicsSharp wrapper, `PhysicsWorld`, `RigidBody` component (box/sphere colliders) +- **Engine.AI** — `AiCommandProcessor` (7 commands), `AiCommandQueue` (thread-safe), MCP HTTP server (Kestrel + SSE), stdio MCP server +- **CortexEngine.App** — Entry point, main loop, scene setup, ImGui debug panels ## Key Conventions @@ -42,25 +39,26 @@ No separate lint command. `dotnet build` with 0 warnings is the standard. Run `d - Input is backend-agnostic via `IInputState` + `Key` enum. No SDL3 types in app code. - Camera controllers use `IInputState`, not `InputMapping` directly. - `RenderBackendFactory.Create(name, width, height, validation)` — backends register by name. -- Custom mesh CPU data uses `NativeMemory.Alloc` (not `Marshal.AllocHGlobal`) to match Raylib's `RL_FREE`. -- `SetShaderValue` uses `float[]` for vectors, not `Vector3`/`Vector4` (marshaling reliability). -- Backface culling disabled (`Rlgl.DisableBackfaceCulling`) for mixed-winding meshes. - -## Current Roadmap - -See `CORTEX_ENGINE_ARCHITECTURE.md` §11 for the full roadmap. Short-term priorities: -- Unit tests -- Texture loading verification -- ImGui integration (medium-term) +- Vulkan types split into `VulkanHandles.cs`, `VulkanEnums.cs`, `VulkanStructs.cs` — struct sizes verified by tests against C headers. +- Push constants: single range, `Vertex|Fragment`, 64B (main pipeline) or 160B (shadow pipeline). +- Light data in SceneUBO (448B): `mat4 vp` + `int numLights` + `int numShadowLights` + `LightData[8]` + `shadowParams[4]` + `ambientColor`. +- Shadow cubemap array: 24 layers (4 lights × 6 faces), `samplerCubeArray` in shader, 16-tap Poisson disk PCF. +- `vkCmdCopyImageToBuffer` for video recording, BGRA format, FFmpeg pipe. +- Matrix convention: `view * proj` (row-major, no `row_major` in GLSL). `proj.M22 *= -1` for Vulkan Y-down. ## Files Not to Edit +- `src/Engine.Graphics.Vulkan/Shaders/*.spv` — compiled SPIR-V, regenerate from `.vert`/`.frag` with `glslangValidator -V` - `CORTEX_ENGINE_ARCHITECTURE.md` — canonical architecture reference, update only when architecture changes -- `src/Engine.Graphics.Vulkan/Shaders/*.spv` — compiled SPIR-V, regenerate from `.vert`/`.frag` with glslangValidator ## Environment - .NET 9 SDK at `$HOME/.dotnet` - `DOTNET_ROOT` and `PATH` must include `$HOME/.dotnet` -- Raylib-cs 8.0.0 (Raylib 6.0 native library bundled in NuGet) -- Display required (X11/Wayland) for Raylib window +- SDL3 (ppy.SDL3-CS 2026.520.0, bundled native libSDL3.so) +- ImGui.NET 1.91.6.1 +- JoltPhysicsSharp 2.21.0 +- Vulkan 1.3+ (validation layers recommended for development) +- FFmpeg (for video recording feature) +- glslangValidator (for shader compilation: `sudo apt install glslang-tools`) +- Display required (X11/Wayland) for Vulkan window diff --git a/CORTEX_ENGINE_ARCHITECTURE.md b/CORTEX_ENGINE_ARCHITECTURE.md index b900a9f..b1756e6 100644 --- a/CORTEX_ENGINE_ARCHITECTURE.md +++ b/CORTEX_ENGINE_ARCHITECTURE.md @@ -1,883 +1,266 @@ -# CORTEX ENGINE — Technical Architecture Manifesto +# CORTEX ENGINE — Technical Architecture -**Project**: AI-Native, Multiplatform 3D Game Engine +**Project**: AI-Native 3D Game Engine **Language**: C# (.NET 9) -**Status**: LOCKED — All dependencies verified as of June 2026 -**Created**: 2026-06-16 +**Render Backend**: Pure P/Invoke Vulkan 1.3 +**Last Updated**: June 2026 --- -## 1. EXECUTIVE SUMMARY +## 1. Overview -Cortex Engine is a 3D game engine built from scratch to provide a Unity-like development experience (GameObject/Component paradigm, Inspector, Hierarchy, Scene View) while being deeply integrated with Multimodal Large Language Models (MMLMs). The engine allows an AI to: +Cortex Engine is a 3D game engine built from scratch with a pure P/Invoke Vulkan 1.3 render backend. No wrapper libraries — direct Vulkan API calls via `vkGetInstanceProcAddr`/`vkGetDeviceProcAddr`. -- **See** the engine state via rendered-frame screenshots, virtual cameras, semantic segmentation maps, and profiler screenshots. -- **Read** the complete ECS world state through native JSON serialization. -- **Modify** the running engine via declarative JSON commands and, in Development Mode, via hot-reloaded C# scripts. +The engine allows an AI to: +- **See** the engine state via rendered-frame screenshots and video recording +- **Read** the complete ECS world state through native JSON serialization +- **Modify** the running engine via declarative JSON commands through MCP (Model Context Protocol) -The architecture prioritizes **production maturity** over experimental technologies: a Render HAL with a Raylib-cs default backend and an optional Vulkan (Silk.NET.Vulkan) backend, Flecs.NET (C# bindings for the C-based Flecs ECS), SDL3-cs (ppy.SDL3-CS), and Hexa.NET.ImGui with a native backend. +### Key Technologies +- **Graphics**: Vulkan 1.3 (dynamic rendering, synchronization2, imageCubeArray) +- **ECS**: Flecs.NET 4.0.4 +- **Windowing**: SDL3 (ppy.SDL3-CS 2026.520.0) +- **Physics**: JoltPhysicsSharp 2.21.0 +- **UI**: ImGui.NET 1.91.6.1 +- **AI**: MCP HTTP server (Kestrel + SSE), 7 tools --- -## 2. EVOLUTION OF ARCHITECTURE DECISIONS - -### 2.1 Initial Discussion - -The project began with an exploration of NVIDIA's open-source physics and graphics ecosystems (PhysX 5, Newton, Warp, Falcor, Flow) and modern middleware for building a custom engine from scratch. Initial candidates included: - -- **Graphics**: WebGPU (wgpu), Diligent Engine, BGFX, NVIDIA Falcor -- **Physics**: Jolt Physics, NVIDIA PhysX 5, Box2D v3 -- **Windowing**: SDL3, GLFW -- **ECS**: Flecs, EnTT, Arch -- **UI**: Dear ImGui - -### 2.2 First Iteration - -The first proposed stack was: - -- C# (.NET 9) + NativeAOT -- Silk.NET + wgpu-native (WebGPU) -- SDL3 (via Silk.NET) -- Flecs ECS -- Dear ImGui with custom WebGPU backend -- Jolt Physics -- Roslyn Compiler API for AI hot-reload - -### 2.3 Identified Risks from Internet Research - -Research revealed critical issues with the first iteration: - -1. **NativeAOT + Roslyn conflict**: NativeAOT explicitly does not support `Assembly.LoadFile()` or `System.Reflection.Emit`. Dynamic C# compilation cannot run inside a NativeAOT binary. This is confirmed by Microsoft Learn documentation. -2. **Silk.NET WebGPU bindings**: The maintainers stated that the official WebGPU examples in Silk.NET are "very bad" and "smoke tests" — not production-ready. -3. **Custom WebGPU ImGui backend**: Would require writing ~400 lines of custom rendering code. -4. **Flecs vs Friflo tradeoff**: Friflo.Engine.ECS is pure managed and faster, but Flecs has mature native C-reflection and built-in JSON serialization that works identically in NativeAOT. - -### 2.4 Final Locked Stack - -The final stack was chosen to eliminate experimental dependencies and maximize production maturity: - -- **C# (.NET 9) with dual-runtime strategy**: JIT for development (Roslyn hot-reload), NativeAOT for release. -- **SDL3-cs**: `ppy.SDL3-CS` — direct, zero-overhead P/Invoke bindings maintained by the osu! team. -- **Render HAL**: `Engine.Graphics` abstraction with pluggable backends. -- **Raylib-cs**: `Raylib-cs` 8.0.0 — default, simple OpenGL-based backend for rapid iteration and screenshot capture. -- **Vulkan**: `Silk.NET.Vulkan` 2.21.0 — optional high-performance backend retained as a reference implementation. -- **MoltenVK**: For macOS/iOS compatibility when using the Vulkan backend. -- **Flecs.NET**: `Flecs.NET.Release` — C# bindings for Flecs with NativeAOT static-link support. -- **ImGui**: `Hexa.NET.ImGui` — ships pre-built SDL3 + Vulkan native backends. -- **Jolt Physics**: `JoltPhysicsSharp` — C# bindings for Jolt Physics, .NET 9/10. -- **Roslyn Compiler API**: For Dev-Mode AI hot-reload. -- **SixLabors.ImageSharp**: For JPEG encoding of diagnostic textures. - ---- - -## 3. TECHNOLOGICAL STACK (Zero-Contradiction Core) - -### 3.1 Language & Runtime - -**C# (.NET 9)** - -The engine uses a **dual-runtime strategy**: - -| Mode | Runtime | AI Scripting | Use Case | -|------|---------|--------------|----------| -| **Development** | .NET 9 JIT | ✅ Roslyn Compiler API + `AssemblyLoadContext` hot-reload | Editor, AI co-development, rapid iteration | -| **Release** | .NET 9 NativeAOT | ❌ JSON commands only | Shipped PC/Mobile/Console/WASM builds | - -**Why this is not a contradiction:** - -NativeAOT cannot JIT new code or load assemblies dynamically. Therefore, the engine builds in two configurations: - -```xml - - - DEV_MODE - - - - - RELEASE_AOT - true - -``` - -All Roslyn and `AssemblyLoadContext` code is wrapped in `#if DEV_MODE`. - -### 3.2 System Layer - -**SDL3 via `ppy.SDL3-CS`** - -- NuGet: `ppy.SDL3-CS` 2026.520.0 -- Direct P/Invoke bindings, zero overhead -- Cross-platform: Windows, Linux, macOS, iOS, Android -- Handles: window creation, input (keyboard, mouse, touch, gamepad, accelerometer), audio, events -- Used by the osu! framework (2K+ stars), battle-tested - -### 3.3 Graphics HAL - -The graphics layer is split into a backend-agnostic **Render HAL** (`Engine.Graphics`) and concrete backend implementations. - -**Core abstraction (`Engine.Graphics`)** - -- `IRenderContext` — backend lifetime, resize, and surface handling. -- `IRenderer` — renders the ECS world and exposes screenshot capture. -- `RenderBackendFactory` — a registry/factory pattern; backend assemblies register themselves. -- The app depends only on these interfaces. - -**Default backend: Raylib-cs** - -- NuGet: `Raylib-cs` 8.0.0 -- Simple, mature OpenGL-based renderer -- Handles window creation, mesh upload, 3D camera, and PNG screenshots internally -- Owns its GLFW window and input via `RaylibWindow` + `RaylibInputState` (no SDL3 dependency) - -**Optional backend: Vulkan via `Silk.NET.Vulkan` — DEFERRED** - -- NuGet: `Silk.NET.Vulkan` 2.21.0 and `Silk.NET.Vulkan.Extensions.KHR` 2.21.0 -- The Vulkan backend compiles and implements the same `IRenderContext` / `IRenderer` HAL interfaces -- Uses `Sdl3Window` internally for Vulkan surface creation (`SDL_Vulkan_CreateSurface`) -- **Status: deferred to long-term backlog.** The backend is kept compilable and architecturally - integrated (via `IWindow`, `IRenderContext`), but is not actively tested or maintained. - The Raylib backend is the primary render path for all current development. -- **Reintegration checklist** (when picked up): - 1. Test `VulkanRenderContext` with the new `IWindow`-based factory signature - 2. Verify `SDL_Vulkan_CreateSurface` works through `IWindow.Handle` - 3. Port improved shading (Fresnel, ACES, gamma, hemisphere ambient) to Vulkan GLSL shaders - 4. Verify custom mesh upload (spheres, grids) works via Vulkan vertex/index buffers - 5. Test screenshot capture via `ScreenshotCapture` with the new frame-deferral logic - -**Why a HAL + Raylib default?** - -- Drastically reduces the code the app, AI commands, and camera tools depend on -- Raylib-cs provides a fast, stable path for screenshots, 3D drawing, and windowing without custom shader/pipeline work -- Vulkan remains available as a high-performance, compute-capable backend for future vision pipelines - -**macOS/iOS path:** - -- When using the Vulkan backend: MoltenVK 1.4 supports Vulkan 1.4 on macOS, iOS, tvOS, visionOS -- `VK_KHR_portability_subset` and `VK_KHR_portability_enumeration` must be enabled -- Loader and MoltenVK libraries must be bundled with the application -- KosmicKrisp (via Mesa 3D) is an emerging alternative for Apple Silicon desktops - -### 3.4 Data Architecture - -**Flecs.NET** - -- NuGet: `Flecs.NET.Debug` (Debug) / `Flecs.NET.Release` (Release) 4.0.4-build.546 -- High-level C# wrapper over the C-based Flecs ECS -- Supports .NET Standard 2.1, .NET 5/6/7/8 -- NativeAOT-compatible via static linking: `true` -- Native C-reflection: `ecs_world_to_json()` serializes the entire ECS world -- Used in AAA engines - -**Why Flecs.NET over pure managed ECS (Friflo, Arch):** - -- Native C-reflection works without C# `System.Reflection` — essential for NativeAOT -- Built-in JSON serialization of the entire world -- Production maturity and industry usage -- Component registration across shared libraries/DLLs -- Automatic archetype management and lockless scheduler - -### 3.5 Editor UI - -**Hexa.NET.ImGui** - -- NuGet: `Hexa.NET.ImGui` (and related backend packages) -- Alternative to ImGui.NET that ships pre-built native backends -- Includes SDL3 + Vulkan backend combinations -- MIT licensed -- Higher performance, reduced startup time -- Eliminates the need to write a custom Vulkan renderer for ImGui - -**Why Hexa.NET.ImGui over ImGui.NET:** - -- `ImGui.NET` does not ship a C# Vulkan renderer out of the box -- The official Vulkan backend is `imgui_impl_vulkan.cpp` (C++), which must be manually compiled and P/Invoked -- Hexa.NET.ImGui bundles the C++ backends as native libraries with C# bindings -- This is the fastest path to a production-ready editor UI - -### 3.6 Physics - -**JoltPhysicsSharp** - -- NuGet: `JoltPhysicsSharp` 2.21.0 -- .NET 9/10 bindings for Jolt Physics -- Cross-platform via `joltc` C wrapper -- Used in Horizon Forbidden West and Death Stranding 2 -- Integrated into Godot 4.4 - -**Note:** Physics module is not part of the foundational MVP but is included in the final project layout. - -### 3.7 Diagnostic Encoding - -**SixLabors.ImageSharp** - -- Pure managed JPEG/PNG encoder -- NativeAOT-compatible -- No native dependencies -- Used to compress Vulkan-rendered RGBA textures into JPEG for MMLM vision input - ---- - -## 4. DATA STRUCTURE & UNITY-LIKE ABSTRACTION - -### 4.1 Core Principle - -The engine exposes a Unity-like API surface (`GameObject`, `AddComponent`, `GetComponent`) while internally storing all data in Flecs components. The `GameObject` wrapper is **never** a place for state. - -### 4.2 GameObject Facade - -```csharp -public readonly struct GameObject -{ - public readonly Entity Entity; - public readonly World World; - - public GameObject(World world, Entity entity) - { - World = world; - Entity = entity; - } - - public void AddComponent(T component) where T : unmanaged - { - Entity.Set(component); - } - - public ref T GetComponent() where T : unmanaged - { - return ref Entity.GetMut(); - } - - public bool HasComponent() where T : unmanaged - { - return Entity.Has(); - } - - public void RemoveComponent() where T : unmanaged - { - Entity.Remove(); - } -} -``` - -### 4.3 Component Definitions - -Components are plain C# structs registered with the Flecs type system: - -```csharp -public struct Transform : IComponent -{ - public Vector3 Position; - public Quaternion Rotation; - public Vector3 Scale; -} - -public struct MeshRef : IComponent -{ - public ulong MeshId; -} - -public struct Camera : IComponent -{ - public Vector3 Position; - public Vector3 Target; - public Vector3 Up; - public float FieldOfView; - public float AspectRatio; - public float NearPlane; - public float FarPlane; -} - -public struct Material : IComponent -{ - public Vector3 Albedo; - public float Roughness; - public float Metallic; - public string? TexturePath; -} - -public struct Light : IComponent -{ - public Vector3 Direction; - public Vector3 Color; - public float Intensity; -} - -public struct SemanticClass : IComponent -{ - public byte ClassId; // 0=environment, 1=enemy, 2=player, 3=interactive, 4=trigger -} -``` - -### 4.4 AI Hot-Reloading (Dev Mode) - -When the AI generates a C# script, the engine: - -1. Receives the script string via the `AiGateway`. -2. Runs a pre-validation pass (syntax check, banned namespace check, unsafe code check). -3. Feeds the script to `Microsoft.CodeAnalysis.CSharp` (Roslyn). -4. Emits the compiled assembly into a `MemoryStream`. -5. Loads the assembly into a **dedicated** `AssemblyLoadContext`. -6. Extracts systems marked with `[Slot("name")]` attribute. -7. Calls `SystemSlotRegistry.HotSwap()` to replace the old system with the new one. -8. Migrates entities using the old component types to the new types. -9. Unloads the old `AssemblyLoadContext`. - -**Important caveat:** Flecs stores component type metadata in its native C memory. If old C# types are still referenced by Flecs, the old `AssemblyLoadContext` cannot be fully unloaded. The migration step must remove old components and re-add them as the new types. - -### 4.5 Rendering & Shading - -The renderer uses a simple forward-lit pipeline that is implemented by each backend behind the HAL: - -- **Vertex format**: position, color, normal. -- **Per-entity**: Mesh + Transform + optional Material. -- **Per-frame constants**: camera position, up to 4 directional lights, ambient color. -- **Per-entity constants**: MVP matrix, material albedo/roughness/metallic, texture use flag. -- **Lighting model**: multiple directional lights with ambient + diffuse + Blinn-Phong specular. -- **Material**: `Material.Albedo` tints vertex color, `Roughness` and `Metallic` control specular falloff and intensity; an optional `TexturePath` enables albedo texture sampling. -- **Vulkan backend**: uses a uniform buffer (descriptor set 0) and push constants; textures are Vulkan images with a combined image sampler (descriptor set 1). -- **Raylib backend**: uses a custom GLSL shader with `materialColor`, `useTexture`, `roughness`, `metallic`, and light arrays. Textures are loaded via `Raylib.LoadTexture` and UVs use world-space XZ. -- **UV mapping**: meshes use world-space XZ as a simple UV mapping for both backends. - -### 4.6 SystemSlotRegistry - -```csharp -public class SystemSlotRegistry -{ - private readonly Dictionary _slots = new(); - private readonly World _world; - - public void Register(string slotName, Entity systemEntity) - { - _slots[slotName] = systemEntity; - } - - public void HotSwap(string slotName, Entity newSystemEntity) - { - if (_slots.TryGetValue(slotName, out var oldSystem)) - { - oldSystem.Destruct(); // Flecs system destructor - } - - _slots[slotName] = newSystemEntity; - } - - public IEnumerable> EnumerateSlots() - { - return _slots; - } -} -``` - ---- - -## 5. MMLM SELF-DIAGNOSTIC CONTEXT LOOP - -### 5.1 Purpose - -The `DiagnosticsManager` captures a multimodal **Diagnostic Payload** that gives the AI full context: - -- **Visual context** — what the engine is currently rendering -- **Structural context** — the exact ECS world state -- **Code context** — current scripts, logs, and stack traces - -### 5.2 Diagnostic Payload Structure - -```csharp -public class DiagnosticPayload -{ - public byte[] VisualJpeg { get; set; } // Semantic scene screenshot - public byte[] ProfilerJpeg { get; set; } // ImGui performance graph - public string WorldJson { get; set; } // Flecs ECS state - public string SystemGraphSvg { get; set; } // System pipeline diagram - public string ConsoleLogs { get; set; } // Recent log tail - public string StackTrace { get; set; } // Exception trace if any - public Dictionary SourceFiles { get; set; } // Active scripts -} -``` - -### 5.3 Capture Flow +## 2. Project Structure ``` -[1] TRIGGER - │ - ├─ User clicks "AI Inspect" in the editor - ├─ Unhandled exception occurs - └─ Automatic capture on frame-time spike - │ -[2] CAPTURE - │ - ├─ [2a] Visual Layer (Vulkan) - │ ├─ Allocate off-screen VkImage (R8G8B8A8_UNORM) - │ ├─ Encode semantic render pass using SemanticClass component - │ ├─ Use Vulkan dynamic rendering (vkCmdBeginRendering / vkCmdEndRendering) - │ ├─ Copy image to host-visible staging buffer (vkCmdCopyImageToBuffer) - │ ├─ Map memory → Span RGBA - │ └─ Encode to JPEG via ImageSharp - │ - ├─ [2b] Visual Layer (Profiler) - │ ├─ Render ImGui profiler graph to the same RTT pipeline - │ └─ Encode to JPEG - │ - ├─ [2c] Structural Layer (Flecs) - │ ├─ Call ecs_world_to_json(world, ¶ms) - │ ├─ Marshal native UTF-8 pointer to managed string - │ └─ Optionally filter by camera frustum - │ - ├─ [2d] Structural Layer (System Graph) - │ ├─ Enumerate SystemSlotRegistry - │ └─ Generate SVG dependency graph - │ - └─ [2e] Code & Log Layer - ├─ Read circular log buffer - ├─ Capture exception stack trace if present - └─ Read active scripts from /projects/ - │ -[3] PACK - │ - └─ JSON envelope with base64-encoded JPEGs: - { - "visual": "", - "profiler": "", - "world": { ... flecs json ... }, - "systems": "", - "logs": "...", - "error": "...", - "sources": { "PlayerController.cs": "..." } - } - │ -[4] DELIVER - │ - ├─ Engine.Broker HTTP POST /diagnostics - │ (External script forwards to MMLM API) - │ - └─ Future: embedded llama.cpp / ONNX Runtime direct inference -``` - -### 5.4 Timing Budget - -| Phase | Target Time | -|-------|-------------| -| Vulkan semantic render pass | 2–5 ms (GPU) | -| Texture readback + JPEG encode | 5–10 ms (CPU) | -| `ecs_world_to_json()` | 1–3 ms (native C) | -| SVG graph generation | <1 ms | -| Log collection | <1 ms | -| **Total** | **~10–20 ms** | - -Non-critical captures can be spread across multiple frames to avoid stuttering. - -### 5.5 Flecs JSON Advantage - -`ecs_world_to_json()` uses **Flecs native C-reflection** instead of C# `System.Reflection`. This means: - -- ✅ Works in NativeAOT Release Mode -- ✅ No P/Invoke overhead for serialization itself -- ✅ Schema-stable output, perfect for MMLM prompts -- ✅ Works with runtime-registered components - ---- - -## 6. AI MUTATION GATEWAY - -### 6.1 Purpose - -The `AiGateway` is the only allowed path for the AI to modify the running engine. It prevents memory corruption, invalid state, and unsafe code execution. - -### 6.2 MCP Server (Dev / Release) - -The engine can expose its AI commands through two MCP transports: - -1. **HTTP MCP server** (Debug/Release): an in-process ASP.NET Core server using `ModelContextProtocol.AspNetCore` with SSE on `http://localhost:/`. Enable with `--mcp-port `. -2. **Stdio MCP server** (Debug/Release): a minimal JSON-RPC server that reads from stdin and writes to stdout. Enable with `--mcp-stdio`. This is the format expected by Claude Desktop and other stdio MCP clients. - -Available tools: - -- `spawn_model` — spawn a named entity from a model file. -- `set_transform` — update entity position, rotation, scale. -- `set_material` — update entity albedo, roughness, metallic, and texture path. -- `delete_entity` — delete an entity by name. -- `list_entities` — list all named entities with a `Transform`. -- `get_world_state` — dump the ECS world as JSON (Transform, Camera, Material, Light, Mesh). -- `capture_screenshot` — capture the current frame, save it as PNG on disk, and return a JSON envelope `{ "path": "...", "base64": "..." }` with the base64-encoded PNG (HTTP/render mode only). - -Commands are queued and executed on the main engine thread so the Flecs world is never touched from a background thread. - -### 6.3 Command Pattern (Works in Both Dev and Release) - -The AI can also issue declarative JSON commands directly: - -```json -{ - "type": "spawn_model", - "name": "Enemy", - "modelPath": "Models/enemy.obj", - "position": [10, 0, 5], - "rotation": [0, 0, 0, 1], - "scale": [1, 1, 1] -} -``` - -Validation steps: - -1. JSON schema validation -2. Type existence check via Flecs reflection -3. Coordinate sanity check (e.g., no NaN, no extreme values) -4. Safe-name check (no `..` in model paths) -5. Queue operation for execution at the next frame boundary - -### 6.4 Scripting Validation (Dev Mode Only) - -Before Roslyn compilation: - -1. **Syntax pre-check**: Parse as C# syntax tree. -2. **Banned symbols check**: Disallow `unsafe`, `Marshal`, `File`, `Process`, `Thread`, `Assembly`, `Reflection.Emit`. -3. **Namespace whitelist**: Allow only `Engine.*`, `System`, `System.Numerics`, `Flecs.NET`. -4. **Reference validation**: Ensure all referenced types exist in the engine API surface. -5. **Sandboxed compilation**: Compile into isolated `AssemblyLoadContext`. - -### 6.5 Release Mode Limitation - -In Release (NativeAOT), the MCP server and ASP.NET Core are excluded. The AI cannot compile new C# code. It can only send JSON commands via `AiCommandProcessor`. This is a deliberate security and stability choice. - ---- - -## 7. LOGICAL PROJECT LAYOUT - -```text -/home/emil/Desktop/Cortex_Engine -├── AGENTS.md # This file -├── CORTEX_ENGINE_ARCHITECTURE.md # Mirror / detailed specification -├── src/ -│ ├── Engine.Core/ -│ │ ├── EngineApp.cs # Entry point, main loop -│ │ ├── Sdl3Window.cs # SDL3 window wrapper -│ │ ├── Timing.cs # DeltaTime, fixed timestep -│ │ ├── InputMapping.cs # Keyboard, mouse, gamepad input - │ │ ├── ICameraController.cs # Camera controller interface - │ │ ├── FreeFlyCameraController.cs # WASD + mouse look camera - │ │ ├── IScreenshotProvider.cs # Async screenshot capture interface -│ │ └── Components/ # Transform, Camera, Light, Material, Mesh -│ │ -│ ├── Engine.Data/ -│ │ ├── GameObject.cs # Thin struct facade -│ │ ├── ComponentTypes.cs # Transform, MeshRef, Camera, SemanticClass -│ │ ├── WorldContext.cs # Flecs world initialization -│ │ └── SystemSlotRegistry.cs # Named system hot-swap registry -│ │ -│ ├── Engine.Graphics/ -│ │ ├── IRenderContext.cs # Backend context abstraction -│ │ ├── IRenderer.cs # ECS world renderer abstraction -│ │ ├── RenderBackendFactory.cs # Backend registry and factory -│ │ └── Loaders/ # ObjLoader, GltfLoader -│ │ -│ ├── Engine.Graphics.Raylib/ -│ │ ├── RaylibBackendRegistrar.cs # Registers the Raylib backend with the factory -│ │ ├── RaylibRenderContext.cs # Raylib window/surface context -│ │ └── RaylibRenderer.cs # Raylib ECS mesh renderer + screenshot capture -│ │ -│ ├── Engine.Graphics.Vulkan/ -│ │ ├── VulkanBackendRegistrar.cs # Registers the Vulkan backend with the factory -│ │ ├── VulkanRenderContext.cs # Vulkan instance, device, surface, swapchain -│ │ ├── VulkanRenderer.cs # Vulkan ECS mesh renderer -│ │ ├── VulkanContext.cs # Device, instance, queues, command pool -│ │ ├── Swapchain.cs # Swapchain + depth buffer -│ │ ├── VulkanPipeline.cs # Graphics pipeline + descriptor layouts -│ │ ├── ScreenshotCapture.cs # Vulkan readback → PNG -│ │ ├── UniformBuffer.cs # Per-frame uniform buffer -│ │ ├── Texture.cs # Vulkan texture (image, view, sampler) -│ │ ├── VertexBuffer.cs # Vertex buffer helpers -│ │ ├── IndexBuffer.cs # Index buffer helpers -│ │ ├── ShaderLoader.cs # Embedded SPIR-V loader -│ │ └── Shaders/ # vertex.vert, fragment.frag, *.spv -│ │ -│ ├── Engine.Diagnostics/ -│ │ ├── DiagnosticsManager.cs # Orchestrator -│ │ ├── FlecsJsonExporter.cs # ecs_world_to_json wrapper -│ │ ├── SystemGraphSvg.cs # SVG dependency graph generator -│ │ ├── Payload.cs # DiagnosticPayload class -│ │ └── LogBuffer.cs # Circular console log buffer -│ │ -│ ├── Engine.AI/ -│ │ ├── AiCommandProcessor.cs # Parses and executes JSON commands -│ │ ├── AiCommandQueue.cs # Thread-safe command queue -│ │ ├── Mcp/ -│ │ │ ├── EngineMcpTools.cs # MCP HTTP tool definitions -│ │ │ └── McpEngineServerHost.cs # In-process MCP HTTP server -│ │ ├── Stdio/ -│ │ │ └── McpStdioServer.cs # Minimal stdio MCP server -│ │ ├── Commands/ # AI command DTOs -│ │ └── Serialization/ # JSON converters for Vector3/Quaternion -│ │ -│ ├── Engine.Editor/ -│ │ ├── ImGuiController.cs # Hexa.NET.ImGui initialization -│ │ ├── HierarchyWindow.cs # Scene hierarchy panel -│ │ ├── InspectorWindow.cs # Component inspector -│ │ ├── AiConsoleWindow.cs # AI Co-Developer panel -│ │ └── ProfilerWindow.cs # Performance graphs -│ │ -│ ├── Engine.Broker/ -│ │ └── DiagnosticsHttpServer.cs # Local HTTP endpoint for AI payloads -│ │ -│ └── Engine.Physics/ -│ └── PhysicsModule.cs # JoltPhysicsSharp integration (future) +src/ +├── Engine.Core/ # Core abstractions, windowing, ECS components +│ ├── IWindow.cs # Backend-agnostic window interface +│ ├── IInputState.cs # Input abstraction +│ ├── Sdl3Window.cs # SDL3 window with Vulkan surface +│ ├── Key.cs # Key enum +│ ├── InputMapping.cs # Input state implementation +│ ├── Timing.cs # Frame timing +│ ├── Vertex.cs # Vertex struct (Position, Color, Normal) +│ ├── FreeFlyCameraController.cs +│ ├── OrbitCameraController.cs +│ └── Components/ +│ ├── Transform.cs # Position, Rotation, Scale +│ ├── Mesh.cs # Vertex[], uint[] indices +│ ├── Material.cs # Albedo, Roughness, Metallic, TexturePath +│ ├── Light.cs # Point/Directional light +│ ├── Camera.cs # Perspective camera +│ └── RigidBody.cs # Physics body (box/sphere) │ -├── projects/ # AI-generated game scripts -│ └── .gitkeep -├── tests/ -│ └── Engine.Tests/ -├── shaders/ -│ ├── semantic.vert.spv -│ └── semantic.frag.spv -└── tools/ - └── svg-generator/ # Optional CLI tools +├── Engine.Graphics/ # Graphics HAL + utilities +│ ├── IRenderContext.cs # Window, CreateRenderer, Resize +│ ├── IRenderer.cs # RenderWorld, Screenshot, ImGui hooks +│ ├── IScreenshotProvider.cs +│ ├── RenderBackendFactory.cs # Register/Create by name +│ ├── ObjLoader.cs # OBJ file parser +│ ├── MeshMath.cs # Face normal computation +│ ├── ProceduralMesh.cs # Sphere, Grid generators +│ └── SceneSerializer.cs # JSON scene save/load +│ +├── Engine.Graphics.Vulkan/ # Pure P/Invoke Vulkan 1.3 +│ ├── VulkanNative.cs # Library loading, vkGetInstanceProcAddr +│ ├── VulkanHandles.cs # Opaque pointer types +│ ├── VulkanEnums.cs # All Vulkan enums/flags +│ ├── VulkanStructs.cs # All Vulkan structs (verified sizes) +│ ├── Vk.cs # Function delegates + loaded pointers +│ ├── VulkanContext.cs # Instance, device, queue, surface, debug +│ ├── VulkanSwapchain.cs # Swapchain, image views, depth buffer +│ ├── VulkanPipeline.cs # Graphics pipeline (PBR + dynamic rendering) +│ ├── VulkanShadowMap.cs # Cubemap array shadows (4 lights × 6 faces) +│ ├── VulkanFrameResources.cs # Command buffers, fences, semaphores, UBO +│ ├── VulkanVertexBuffer.cs # Staging → device-local vertex buffer +│ ├── VulkanIndexBuffer.cs # Staging → device-local index buffer +│ ├── VulkanImGui.cs # ImGui font atlas + pipeline + render +│ ├── VulkanRenderer.cs # Main renderer: shadow passes + main pass +│ ├── VulkanRenderContext.cs +│ ├── VulkanBackendRegistrar.cs +│ └── Shaders/ +│ ├── triangle.vert/frag # PBR + multi-light + shadow sampling +│ ├── shadow.vert/frag # Depth-only shadow pass +│ └── imgui.vert/frag # ImGui rendering +│ +├── Engine.Physics/ # Jolt physics wrapper +│ └── PhysicsWorld.cs # Body creation, update, sync transforms +│ +├── Engine.AI/ # AI command system + MCP server +│ ├── AiCommandProcessor.cs # 7 commands (spawn, transform, material, etc.) +│ ├── AiCommandQueue.cs # Thread-safe queue (MCP → main thread) +│ ├── Commands/ # Command DTOs +│ ├── Mcp/ # HTTP MCP server (Kestrel + SSE) +│ └── Stdio/ # Stdio MCP server +│ +└── CortexEngine.App/ # Entry point + └── Program.cs # Main loop, scene, ImGui panels, video recording ``` --- -## 8. FOUNDATIONAL MVP — COMPLETED +## 3. Vulkan Backend -### Step 1: Window + Render HAL + Raylib Backend — DONE +### 3.1 Initialization -- `IWindow` / `IInputState` / `Key` abstractions in `Engine.Core` -- `Sdl3Window` (SDL3) and `RaylibWindow` (GLFW) both implement `IWindow` -- `RenderBackendFactory` — backends register by name, each owns its window -- `RaylibRenderer` — custom GLSL shader, PBR-like lighting, screenshots -- Vulkan backend compiles but is **deferred** (see §3.3) +1. **Load `libvulkan.so.1`** via `NativeLibrary.Load()` +2. **`vkGetInstanceProcAddr`** — only directly-loaded function +3. **Create instance** with SDL3 extensions + `VK_EXT_debug_utils` (if validation available) +4. **Pick physical device** — prefer `DiscreteGpu` +5. **Create logical device** with features: + - `VK_KHR_swapchain` extension + - `VkPhysicalDeviceDynamicRenderingFeatures` (dynamic rendering) + - `VkPhysicalDeviceSynchronization2Features` (sync2) + - `VkPhysicalDeviceFeatures.imageCubeArray = VK_TRUE` (cubemap array shadows) +6. **Create surface** via `SDL_Vulkan_CreateSurface()` -### Step 2: Flecs World + Components + Camera Controllers — DONE - -- `World` (Flecs.NET) with `Transform`, `Mesh`, `Material`, `Light`, `Camera` components -- `FreeFlyCameraController` and `OrbitCameraController` using `IInputState` + `Key` enum -- Procedural mesh generation: `CreateGridMesh`, `CreateSphereMesh` - -### Step 3: AI Bridge + MCP Server — DONE - -- `AiCommandProcessor` — 7 commands: spawn_model, set_transform, set_material, delete_entity, list_entities, capture_screenshot, get_world_state -- HTTP MCP server (SSE, `--mcp-port`) and stdio MCP server (`--mcp-stdio`) -- Screenshot capture with 10-frame warm-up for stable GPU output - ---- - -## 9. VERIFIED DEPENDENCY TABLE - -| Component | Package | Version | .NET | AOT | WASM | Mobile | Status | -|-----------|---------|---------|------|-----|------|--------|--------| -| SDL3 | `ppy.SDL3-CS` | 2026.520.0 | 9/10 | ✅ | ✅ | ✅ | Production | -| Vulkan | `Silk.NET.Vulkan` | 2.21.0 | 9/10 | ✅ | ❌ | MoltenVK | Production | -| ImGui | `Hexa.NET.ImGui` | latest | 9/10 | ✅ | ✅ | ✅ | Production | -| ECS | `Flecs.NET.Release` | 4.0.4-build.546 | 8/9 | ✅* | ✅ | ✅ | Production | -| Model loading | `SharpGLTF.Core` | 1.0.6 | 9/10 | ✅ | ✅ | ✅ | Production | -| AI bridge | `ModelContextProtocol` | 1.4.0 | 8/9 | ❌† | ✅ | ✅ | Production | -| Screenshot | `SixLabors.ImageSharp` | 3.1.11 | 9/10 | ✅ | ✅ | ✅ | Production | -| Physics | `JoltPhysicsSharp` | 2.21.0 | 9/10 | ✅ | ❌ | ✅ | Production | -| JPEG | `SixLabors.ImageSharp` | latest | 9/10 | ✅ | ✅ | ✅ | Production | - -\* Via `true` - -† MCP server is excluded from `ReleaseAOT` because it depends on ASP.NET Core. JSON-only AI commands still work in AOT via `AiCommandProcessor`. - ---- - -## 10. KNOWN RISKS & MITIGATIONS - -| Risk | Impact | Mitigation | -|------|--------|------------| -| NativeAOT + Roslyn conflict | High | Dual-runtime strategy: JIT for dev, AOT for release | -| Flecs AssemblyLoadContext leak | Medium | Type migration before unloading old context | -| MoltenVK limitations | Medium | Use Vulkan 1.3 baseline + portability subset; test on Apple hardware early | -| Vulkan verbosity | Medium | Build a high-level renderer abstraction; let the AI generate systems, not raw Vulkan | -| Hexa.NET.ImGui version drift | Low | Pin version; fork if necessary | -| JoltPhysicsSharp mobile perf | Low | Profile on target devices; use Jolt's SIMD paths | - ---- - -## 11. CURRENT ROADMAP (Post-MVP) - -### Completed - -- [x] Modular window/input HAL (`IWindow`, `IInputState`, `Key` enum) -- [x] Raylib backend as primary render path (GLFW window, no SDL3 dependency) -- [x] PBR-like shading: Fresnel (Schlick), hemisphere ambient, ACES tonemapping, gamma correction -- [x] Procedural mesh generation (spheres, grids) with correct memory management -- [x] FreeFly + Orbit camera controllers with inverted-yaw and strafe fixes -- [x] MCP server (HTTP + stdio) with 7 AI commands -- [x] Demo scene with cubes + spheres showcasing different materials - -### Short-term (next) - -- [x] Texture loading in RaylibRenderer (`SetMaterialUniforms` now loads/binds textures) -- [x] Fix `demo.png` screenshot timing (moved to main loop with frame warm-up) -- [x] Unit tests — 60 tests covering ObjLoader, camera controllers, AiCommandProcessor, RenderBackendFactory, Timing, ProceduralMesh, MeshMath, Transform, Camera, Material, Mesh, Light -- [x] `AGENTS.md` — created for opencode integration - -### Medium-term - -- [x] Dear ImGui integration (rlImgui-cs + ImGui.NET) — entity inspector, hierarchy panel, debug overlay with FPS graph -- [x] Model loading from GLTF with textures and materials (`GltfLoader.LoadWithMaterials` extracts PBR albedo, roughness, metallic, base color texture) -- [x] Scene serialization / deserialization (`SceneSerializer` — save/load named entities with Transform, Material, Light, Camera to/from JSON) -- [x] Multi-light shadow mapping — attempted but Raylib's DrawModelEx doesn't support multi-texture-unit binding. Shadow code removed. Deferred to when using rlgl directly or Vulkan backend. - -### Long-term (backlog) - -- [ ] **Vulkan backend reintegration** — see §3.3 checklist. Compiles but untested. - Kept architecturally compatible via `IWindow` / `IRenderContext` / `IRenderer`. - Deferred because Raylib covers all current needs with far less complexity. -- [x] Physics (JoltPhysicsSharp 2.21.0) — Engine.Physics project, PhysicsWorld wrapper, - RigidBody ECS component, dynamic boxes/spheres with gravity, static floor, transform sync -- [ ] AI hot-reload of C# scripts (Roslyn — conflicts with NativeAOT) -- [ ] Semantic segmentation maps for MMLM vision input - ---- - -## 12. PROMPT ENGINEERING FOR AI CODING - -When generating code with an MMLM for this engine, always include this context header: +### 3.2 Frame Loop ``` -You are coding for Cortex Engine, a C# (.NET 9) AI-Native multiplatform 3D game engine. - -Stack: -- C# .NET 9 with dual-runtime: JIT (Debug) for Roslyn hot-reload, NativeAOT (ReleaseAOT) for JSON-only AI commands -- SDL3-cs (ppy.SDL3-CS) for windowing and input -- Silk.NET.Vulkan for graphics -- Flecs.NET for ECS -- Hexa.NET.ImGui for editor UI -- JoltPhysicsSharp for physics -- ModelContextProtocol for AI tool integration - -Rules: -1. All state lives in ECS components (Transform, Camera, Light, Mesh, Material). -2. All AI mutations go through Engine.AI (AiCommandProcessor / MCP tools). -3. All Dev-Mode AI scripts must be AOT-compatible and avoid unsafe, Reflection.Emit, Assembly.Load, File I/O. -4. All systems are registered in SystemSlotRegistry and tagged with [Slot("name")]. -5. Use Roslyn syntax trees for validation before compilation. -6. Prefer Flecs native reflection (ecs_world_to_json) over C# reflection. -7. Keep modules isolated; do not create circular dependencies between Engine.Core, Engine.Graphics, Engine.AI. - -Current file context: [insert path here] +Each frame: + 1. WaitFrame (fence) + 2. Read captured frame buffer (if recording video) + 3. AcquireNextImageKHR + 4. Begin command buffer + 5. Shadow passes (numShadowLights × 6 faces): + - Transition shadow cubemap array → ColorAttachment + DepthAttachment + - For each shadow light, for each face: + - Begin rendering (color R32_SFLOAT + depth D32_SFLOAT) + - Bind shadow pipeline (depth-only, CULL_NONE, depth bias) + - Push constants: model + lightViewProj + lightPos + shadowParams (160B) + - Draw all shadow-casting objects + - End rendering + - Transition shadow cubemap array → ShaderReadOnlyOptimal + 6. Main pass: + - Transition swapchain image → ColorAttachmentOptimal + - Transition depth image → DepthStencilAttachmentOptimal + - Begin rendering (color B8G8R8A8_UNORM + depth D32_SFLOAT) + - Bind pipeline, descriptor sets (SceneUBO + shadowCubeArray) + - For each object: push constants (model 64B), draw indexed + - [If recording] End rendering, capture frame (vkCmdCopyImageToBuffer) + - [If recording] Begin new rendering with loadOp=LOAD for ImGui + - ImGui render + - End rendering + - Transition swapchain → PresentSrcKHR + 7. End command buffer + 8. QueueSubmit2 (sync2) + 9. QueuePresentKHR ``` +### 3.3 SceneUBO Layout (464B) + +``` +Offset 0: mat4 vp (64B) +Offset 64: int numLights (4B) +Offset 68: int numShadowLights (4B) +Offset 72: vec2 padding (8B) +Offset 80: LightData lights[8] (256B) — 2 × vec4 per light +Offset 336: vec4 shadowParams[4] (64B) — bias, sampleRadius, farPlane +Offset 400: vec4 ambientColor (16B) +Total: 416B → padded to 464B (align 64) +``` + +### 3.4 Push Constants + +**Main pipeline**: `mat4 model` (64B), `Vertex|Fragment` +**Shadow pipeline**: `mat4 model` + `mat4 lightViewProj` + `vec4 lightPos` + `vec4 shadowParams` (160B), `Vertex|Fragment` + +### 3.5 Shadow Mapping + +- **Cubemap array**: 1 image, 24 layers (4 lights × 6 faces), `CubeCompatible` flag +- **Color attachment**: R32_SFLOAT (stores linear distance / farPlane) +- **Depth attachment**: D32_SFLOAT (depth test only) +- **Sampling**: `samplerCubeArray`, `texture(shadowArray, vec4(dir, lightIndex))` +- **Filtering**: 16-tap Poisson disk PCF, slope-dependent bias +- **Vulkan cubemap conventions**: up=(0,-1,0) for X/Z faces, up=(0,0,1) for +Y, up=(0,0,-1) for -Y + +### 3.6 PBR Shading + +Cook-Torrance BRDF: +- **D**: Trowbridge-Reitz GGX distribution +- **G**: Smith geometry with Schlick-GGX +- **F**: Schlick Fresnel approximation +- **Tonemapping**: ACES filmic +- **Gamma**: 2.2 correction +- Multi-light loop in fragment shader + --- -## 13. NEXT DECISION POINTS +## 4. AI/MCP Integration -1. Add ImGui editor UI (`Hexa.NET.ImGui`) for scene hierarchy and inspector. -2. Add physics integration (`JoltPhysicsSharp`) with rigid bodies and colliders. -3. Implement semantic segmentation render pass for AI vision. -4. Add audio module (`NAudio` or `OpenAL` bindings). -5. Add networking / multiplayer foundation. +### 7 MCP Tools + +| Tool | Description | +|---|---| +| `spawn_model` | Create entity with mesh + optional physics + shape (cube/sphere) | +| `set_transform` | Update position/rotation/scale by name | +| `set_material` | Update albedo/roughness/metallic/texture | +| `delete_entity` | Remove entity by name | +| `list_entities` | List all named entities | +| `get_world_state` | Full JSON state dump | +| `capture_screenshot` | Request screenshot | + +### Threading + +- MCP server runs on `Task.Run` (Kestrel thread pool) +- Commands enqueued via `AiCommandQueue` (thread-safe `ConcurrentQueue`) +- Main thread calls `ProcessPending()` before render each frame +- `CompletePendingScreenshots()` after render --- -## 14. RUNTIME NOTES & CRITICAL CONTEXT +## 5. Physics -### 14.1 Building & Running - -```bash -export DOTNET_ROOT="$HOME/.dotnet" -export PATH="$DOTNET_ROOT:$PATH" -export DISPLAY=:0 -dotnet build CORTEX_ENGINE.sln -c Debug - -# Convenience script (handles DOTNET_ROOT/PATH/DISPLAY automatically): -./scripts/run.sh - -# Or run directly: -dotnet run --project src/CortexEngine.App/CortexEngine.App.csproj -``` - -- `RuntimeIdentifier=linux-x64` is required in Debug to use the bundled native `libSDL3.so` from `ppy.SDL3-CS` (system `libSDL3.so.3.4.2` is ABI-incompatible). -- AOT builds: `dotnet build CORTEX_ENGINE.sln -c ReleaseAOT`. - -### 14.2 CLI Arguments - -- `--mcp-port ` — start the HTTP MCP server on `http://localhost:/` (SSE). -- `--mcp-stdio` — run the headless stdio MCP server for Claude Desktop / other stdio clients. -- `--camera-tour` — capture screenshots from predefined poses and exit. -- `--test-scene` — enable a calibration scene with colored cubes at known world positions and run a camera tour. Useful for visually verifying perspective and camera movement. -- Any other positional argument is treated as a model path (`.obj`, `.gltf`, `.glb`). - -### 14.3 Convenience Scripts - -| Script | Purpose | -|--------|---------| -| `./scripts/run.sh` | Run the engine; passes all arguments to the app (e.g., `./scripts/run.sh --mcp-port 5000`). | -| `./scripts/start_mcp_engine.sh ` | Run the engine with MCP enabled on the given port (default 5000). | - -### 14.4 Graphics Backends - -**Default backend: Raylib-cs** - -- The app calls `RenderBackendFactory.Create("raylib", width, height, enableValidation: false)`. -- `RaylibRenderContext` creates a `RaylibWindow` (GLFW) and `RaylibRenderer` handles the frame. -- `RaylibRenderer` uploads `Mesh` data to GPU via `LoadModelFromMesh`, sets a custom GLSL 330 core - shader with Fresnel, ACES tonemapping, gamma correction, hemisphere ambient, and up to 4 - directional lights. Renders the ECS world via `DrawModelEx`. -- Backface culling is disabled (`Rlgl.DisableBackfaceCulling`) for compatibility with mixed-winding meshes. -- Screenshots are captured via `Raylib.LoadImageFromScreen` with a 10-frame warm-up delay. -- Custom mesh CPU data is allocated via `NativeMemory.Alloc` (matching Raylib's `RL_FREE` allocator) - and kept alive until `UnloadModel` — freeing early caused broken large meshes (spheres, grids). - -**Vulkan backend (DEFERRED — not actively tested)** - -- Compiles and registers via `VulkanBackendRegistrar`, but is not the active render path. -- Uses `Sdl3Window` internally for `SDL_Vulkan_CreateSurface`. -- Pipeline layout uses **two descriptor sets**: set 0 = per-frame uniform buffer (camera + lights), - set 1 = per-entity combined image sampler. -- Push constants: 96 bytes (`mat4 mvp` + material albedo/roughness/metallic + texture flag + padding). -- Shaders are compiled with `glslangValidator`: - ```bash - /tmp/glslang/bin/glslangValidator -V src/Engine.Graphics.Vulkan/Shaders/vertex.vert -o src/Engine.Graphics.Vulkan/Shaders/vertex.spv - /tmp/glslang/bin/glslangValidator -V src/Engine.Graphics.Vulkan/Shaders/fragment.frag -o src/Engine.Graphics.Vulkan/Shaders/fragment.spv - ``` -- See §3.3 for the reintegration checklist. - -### 14.5 Input - -- Input is backend-agnostic via `IInputState` + `Key` enum (defined in `Engine.Core`). -- **Raylib backend**: `RaylibInputState` polls Raylib's input functions directly (no SDL3). -- **Vulkan backend** (deferred): `Sdl3Window` + `InputMapping` polls SDL3 events. -- **FreeFly camera** (default): `WASD` — move, `Q`/`E` — down/up, `Shift` — boost, right-click + mouse — look. -- **Orbit camera** (toggle with `F`): right-click + mouse — orbit target `(0, 0.5, 0)`, wheel — zoom, `WASD`/`Q`/`E`/`Shift` — move target. -- `ESC` — exit. -- Default camera: `(0, 0.75, -30)`, target `(0, 0.5, 0)`, FOV 15° (vertical), near 0.1, far 100. - -### 14.6 MCP Client Config - -Sample Claude Desktop config (`claude_desktop_config.json`): - -```json -{ - "mcpServers": { - "cortex-engine": { - "command": "dotnet", - "args": [ - "run", - "--project", - "/home/emil/Desktop/Cortex_Engine/src/CortexEngine.App/CortexEngine.App.csproj", - "--", - "--mcp-stdio" - ], - "env": { - "DOTNET_ROOT": "/home/emil/.dotnet", - "PATH": "/home/emil/.dotnet:/usr/bin:/bin" - } - } - } -} -``` - -For the HTTP MCP server, use the `--mcp-port` argument and connect an SSE MCP client. - -### 14.7 Process Cleanup - -Background `dotnet run` processes may leave the apphost running. Kill them with: - -```bash -ps -C CortexEngine.App -o pid= | xargs -r kill -9 -``` +- JoltPhysicsSharp 2.21.0 +- `PhysicsWorld`: gravity (0, -9.81, 0), 2 object layers (NonMoving, Moving) +- `RigidBody` component: DynamicBox, DynamicSphere, StaticBox, StaticPlane +- Lazy initialization: `IsInitialized` flag, bodies created on first frame +- `SyncTransforms`: pulls Jolt positions/rotations back into ECS Transforms +- Pause/Resume via `physicsEnabled` flag +- Reset Scene: `RemoveBody` before `Destruct` to clean Jolt references --- -*This document is the canonical architecture reference for Cortex Engine. Any changes to stack, project layout, or core data flow must be reflected here before implementation.* +## 6. ImGui + +- Separate pipeline with alpha blending, no depth write +- Font atlas uploaded to R32_SFLOAT VkImage via staging buffer +- Mouse input from `IInputState` fed to `ImGui.GetIO()` each frame +- Debug panel: FPS, camera, entity count, physics toggle, reset scene +- Shadow & Light panel: per-light intensity/range/RGB, shadow bias/radius/farplane, ambient RGB +- Video recording panel: Start/Stop buttons + +--- + +## 7. Video Recording + +- `vkCmdCopyImageToBuffer` copies swapchain image to HOST_VISIBLE buffer +- BGRA pixel data piped to FFmpeg via stdin +- FFmpeg: `rawvideo bgra → libx264 yuv420p`, 60 FPS input, 30 FPS output +- Capture before ImGui (video without UI) +- Deferred read: buffer read at start of next frame (after fence wait) + +--- + +## 8. Content + +| File | Description | +|---|---| +| `cube.obj` | Unit cube, CCW winding | +| `sphere.obj` | UV sphere 24×16, CCW winding | +| `torusknot.obj` | Torus knot (3,2), 4800 verts, smooth normals | +| `torus.obj` | Donut, 24×16 segments, CCW winding | +| `pyramid.obj` | 4-sided pyramid, CCW winding | +| `diamond.obj` | Octahedron (8 faces), CCW winding | +| `cone.obj` | 24-sector cone with bottom cap, CCW winding | + +--- + +## 9. Testing + +227 xUnit tests: +- **VulkanStructSizeTests** (47): C# struct sizes match C Vulkan headers +- **VulkanEnumValueTests** (40+): sType, format, layout, topology, sync2 flags +- **VertexLayoutTests** (5): Vertex struct size (36B), field offsets +- **ObjLoaderTests** (8): OBJ parsing, face normals, winding +- **ObjLoaderFaceNormalTests** (5): Face normal computation +- **ShadowMapTests** (10): Face directions, FOV, up vectors, far plane +- **ShadowShaderTests** (5): SPIR-V shader existence +- **CameraTests, TransformTests, TimingTests, CameraControllerTests, AiCommandProcessorTests, RenderBackendFactoryTests, MeshMathAndProceduralTests, SceneSerializerTests**: Core functionality diff --git a/README.md b/README.md new file mode 100644 index 0000000..29dc73f --- /dev/null +++ b/README.md @@ -0,0 +1,75 @@ +# Cortex Engine + +AI-Native 3D game engine with pure P/Invoke Vulkan 1.3 render backend. + +## Features + +- **Vulkan 1.3** — dynamic rendering, synchronization2, pure P/Invoke (no wrapper libraries) +- **PBR Shading** — Cook-Torrance BRDF, ACES tonemapping, multi-light support +- **Cubemap Array Shadows** — omnidirectional shadows from multiple point lights, 16-tap Poisson disk PCF +- **Jolt Physics** — rigid body dynamics, box/sphere colliders, gravity +- **ImGui** — debug overlay, light/shadow/physics controls +- **AI/MCP** — 7 tools (spawn, transform, material, delete, list, world state, screenshot) via HTTP/SSE +- **Video Recording** — FFmpeg pipe, 60 FPS capture, 30 FPS output +- **ECS** — Flecs.NET with Transform, Mesh, Material, Light, Camera, RigidBody components +- **SDL3 Window** — cross-platform, Vulkan surface + +## Quick Start + +```bash +# Build +dotnet build CORTEX_ENGINE.sln -c Debug + +# Run +./scripts/run.sh + +# Run with AI/MCP server +./scripts/run.sh -- --mcp-port 5000 + +# Run tests +dotnet test tests/Engine.Tests/Engine.Tests.csproj -c Debug +``` + +## Controls + +- **WASD** — move camera +- **Right-click + drag** — look around +- **Q/E** — down/up +- **Shift** — speed boost +- **ESC** — quit + +## ImGui Panels + +- **Cortex Engine Debug** — FPS, camera, entity count, physics toggle, reset scene +- **Shadow & Light Parameters** — per-light intensity/range/RGB, shadow bias/radius/farplane, ambient RGB +- **Video Recording** — start/stop recording to MP4 + +## MCP Tools + +Connect via `http://localhost:5000/` (SSE): + +| Tool | Description | +|---|---| +| `spawn_model` | Create object (cube/sphere, optional physics) | +| `set_transform` | Move/rotate/scale by name | +| `set_material` | Change color/roughness/metallic | +| `delete_entity` | Remove object | +| `list_entities` | List all objects | +| `get_world_state` | Full JSON state | +| `capture_screenshot` | Request screenshot | + +## Requirements + +- .NET 9 SDK +- Vulkan 1.3+ drivers +- SDL3 (bundled via NuGet) +- FFmpeg (for video recording) +- glslangValidator (for shader recompilation) + +## Architecture + +See [CORTEX_ENGINE_ARCHITECTURE.md](CORTEX_ENGINE_ARCHITECTURE.md) for full technical docs. + +## License + +All rights reserved. diff --git a/VULKAN_IMPLEMENTATION_PLAN.md b/VULKAN_IMPLEMENTATION_PLAN.md index 24bf981..f348a36 100644 --- a/VULKAN_IMPLEMENTATION_PLAN.md +++ b/VULKAN_IMPLEMENTATION_PLAN.md @@ -1,729 +1,29 @@ # CORTEX ENGINE — VULKAN RENDERER IMPLEMENTATION PLAN -## Project State (June 2026) - -### What Exists -- **Engine.Core** — Sdl3Window (SDL3, Vulkan surface ready), IWindow, IInputState, Key enum, InputMapping, - camera controllers (FreeFly, Orbit), components (Transform, Mesh, Material, Light, Camera, RigidBody), - Vertex struct (Position, Color, Normal — 9 floats), Timing, IScreenshotProvider -- **Engine.Graphics** — Restored minimal interfaces: IRenderContext, IRenderer, RenderBackendFactory, - IScreenshotProvider, SceneSerializer, MeshMath, ProceduralMesh, Loaders/ObjLoader -- **Engine.Physics** — JoltPhysicsSharp 2.21.0, PhysicsWorld wrapper, RigidBody component -- **Engine.AI** — AiCommandProcessor (7 commands), MCP HTTP + stdio servers, AiCommandQueue -- **CortexEngine.App** — main loop (broken, references deleted graphics projects) -- **tests/Engine.Tests** — 66 tests (broken, reference Engine.Graphics) -- **Content/** — cube.obj, torusknot.obj, checker.png - -### What Was Deleted -- Engine.Graphics.Raylib -- Engine.Graphics.OpenTK -- Engine.Graphics.Vulkan (Silk.NET version — all previous PBR/ImGui/mesh/screenshot code gone) - -### Environment -- .NET 9 SDK at `$HOME/.dotnet` -- Vulkan 1.4.329, NVIDIA RTX 2080 Ti, validation layers available -- SDL3 (ppy.SDL3-CS 2026.520.0) — window + Vulkan surface -- glslangValidator: check availability (`glslangValidator --version`); fallback: `glslc` -- Linux (X11), cross-platform target (Windows: `vulkan-1.dll`, Linux: `libvulkan.so.1`) - ---- - -## Key Architecture Decisions - -| Decision | Choice | Rationale | -|---|---|---| -| Vulkan version | **1.3** | Dynamic rendering (no VkRenderPass/VkFramebuffer), synchronization2, extended dynamic state. All modern GPUs (2022+) support it. | -| Wrapper libraries | **None** | Pure P/Invoke to `libvulkan.so.1` / `vulkan-1.dll`. No Silk.NET, Vortice, OpenTK. | -| Windowing | **SDL3** (ppy.SDL3-CS) | Already integrated, Vulkan surface support built in. | -| Type organisation | **Multiple files** | `VulkanHandles.cs`, `VulkanEnums.cs`, `VulkanStructs.cs` — easier to maintain. | -| Debug | **Full debug messenger** | `VK_EXT_debug_utils` with callback printing validation messages to console (Debug only). | -| Memory | **Staging buffer from start** | Staging buffer (HOST_VISIBLE) → command buffer copy → device-local vertex buffer. Correct pattern from day one. | -| Frame loop | **Re-record every frame** | Vulkan Guide recommends fresh command buffers per frame over reuse. Simpler, no cache invalidation logic. | -| Semaphore indexing | **Per-swapchain-image for submit** | Critical: submit semaphores indexed by swapchain image index, NOT frame-in-flight index. (Vulkan Guide §swapchain_semaphore_reuse) | -| Render pass | **Dynamic rendering** | `vkCmdBeginRendering` / `vkCmdEndRendering` (Vulkan 1.3). No VkRenderPass or VkFramebuffer objects. | -| Synchronisation API | **synchronization2** | `VkImageMemoryBarrier2`, `vkCmdPipelineBarrier2` — cleaner, 64-bit flags. (Vulkan 1.3) | - ---- - -## Implementation Phases - -### Phase 1: Vulkan P/Invoke Foundation - -Create `src/Engine.Graphics.Vulkan/` with the following files: - -#### 1.1 `VulkanNative.cs` -- Load `libvulkan.so.1` (Linux) / `vulkan-1.dll` (Windows) via `NativeLibrary.Load()` -- Export `vkGetInstanceProcAddr` delegate — the only directly-loaded function -- Helper: `GetExport(string name)` for static exports -- Helper: `ToUtf8Terminated(string)` for passing string names to Vulkan - -#### 1.2 `VulkanHandles.cs` -Opaque pointer handles (all are `nint` / `ulong`): -``` -VkInstance, VkPhysicalDevice, VkDevice, VkQueue, -VkCommandPool, VkCommandBuffer, -VkSwapchainKHR, VkSurfaceKHR, -VkImage, VkImageView, -VkBuffer, VkDeviceMemory, -VkShaderModule, VkPipelineLayout, VkPipeline, -VkSemaphore, VkFence, -VkDebugUtilsMessengerEXT, -VkDescriptorSetLayout, VkDescriptorPool, VkDescriptorSet -``` -Each defined as `struct VkXxx { public nint Handle; }` or `using VkXxx = System.IntPtr;` - -#### 1.3 `VulkanEnums.cs` -All enums needed for triangle + future expansion: -- `VkResult` — Success=0, NotReady, Timeout, Incomplete, ErrorOutOfDateKHR, SuboptimalKHR, ErrorSurfaceLostKHR, ... -- `VkStructureType` — ApplicationInfo=0, InstanceCreateInfo=1, DeviceQueueCreateInfo=2, DeviceCreateInfo=3, ... -- `VkFormat` — Undefined=0, R8G8B8A8Unorm=37, B8G8R8A8Unorm=44, R8G8B8A8Srgb=43, B8G8R8A8Srgb=50, R32G32Sfloat=103, R32G32B32Sfloat=106, R32G32B32A32Sfloat=109, D32Sfloat=126, ... -- `VkColorSpaceKHR` — SrgbNonlinear=0 -- `VkPresentModeKHR` — Immediate=0, Mailbox=1, Fifo=2, FifoRelaxed=3 -- `VkImageUsageFlags` — TransferSrc, TransferDst, ColorAttachment, ... -- `VkImageLayout` — Undefined=0, General=1, ColorAttachmentOptimal=2, TransferSrcOptimal=6, TransferDstOptimal=7, PresentSrcKHR=1000001002, ... -- `VkImageAspectFlags` — Color=1, Depth=2 -- `VkAttachmentLoadOp` — Load=0, Clear=1, DontCare=2 -- `VkAttachmentStoreOp` — Store=0, DontCare=1 -- `VkSharingMode` — Exclusive=0, Concurrent=1 -- `VkCompositeAlphaFlagsKHR` — Opaque=1, ... -- `VkSurfaceTransformFlagsKHR` — Identity=1, ... -- `VkPrimitiveTopology` — PointList=0, LineList=1, TriangleList=3, ... -- `VkPolygonMode` — Fill=0, Line=1, Point=2 -- `VkCullModeFlags` — None=0, Front=1, Back=2, FrontAndBack=3 -- `VkFrontFace` — CounterClockwise=0, Clockwise=1 -- `VkBlendFactor` — Zero=0, One=1, SrcAlpha=6, OneMinusSrcAlpha=7, ... -- `VkBlendOp` — Add=0, ... -- `VkColorComponentFlags` — R=1, G=2, B=4, A=8 -- `VkShaderStageFlags` — Vertex=1, Fragment=0x10, AllGraphics=0x1F -- `VkPipelineStageFlags2` — None=0, TopOfPipe=1, ColorAttachmentOutput=0x400, AllGraphics=0x8000, Transfer=0x10000, ... -- `VkAccessFlags2` — None=0, ColorAttachmentWrite=0x400, TransferWrite=0x1000, ... -- `VkDynamicState` — Viewport=0, Scissor=1, ... -- `VkCommandBufferLevel` — Primary=0, Secondary=1 -- `VkCommandBufferUsageFlags` — OneTimeSubmit=1, ... -- `VkFenceCreateFlags` — Signaled=1 -- `VkMemoryPropertyFlags` — DeviceLocal=1, HostVisible=2, HostCoherent=4, HostCached=8 -- `VkBufferUsageFlags` — TransferSrc=1, TransferDst=2, VertexBuffer=0x80, IndexBuffer=0x40, UniformBuffer=0x10 -- `VkQueueFlags` — Graphics=1, Compute=2, Transfer=4 -- `VkPhysicalDeviceType` — Other=0, IntegratedGpu=1, DiscreteGpu=2, ... -- `VkSampleCountFlags` — Count1=1 -- `VkImageViewType` — Type2D=1 -- `VkComponentSwizzle` — Identity=0, ... -- `VkBool32` — False=0, True=1 -- `VkRenderingFlags` — None=0, ContentsSecondaryCommandBuffers=1 -- `VkPipelineBindPoint` — Graphics=0, Compute=1 -- `VkDescriptorType` — UniformBuffer=6, StorageBuffer=7, CombinedImageSampler=0, ... -- `VkDescriptorPoolCreateFlags` — FreeDescriptorSet=1, ... - -#### 1.4 `VulkanStructs.cs` -All structs with `LayoutKind.Sequential`: -- `VkApplicationInfo` — sType, pNext, pApplicationName, applicationVersion, pEngineName, engineVersion, apiVersion -- `VkInstanceCreateInfo` — sType, pNext, flags, pApplicationInfo, enabledLayerCount, ppEnabledLayerNames, enabledExtensionCount, ppEnabledExtensionNames -- `VkDebugUtilsMessengerCreateInfoEXT` — sType, pNext, flags, messageSeverity, messageType, pfnUserCallback, pUserData -- `VkDeviceQueueCreateInfo` — sType, pNext, flags, queueFamilyIndex, queueCount, pQueuePriorities -- `VkDeviceCreateInfo` — sType, pNext, flags, queueCreateInfoCount, pQueueCreateInfos, enabledLayerCount, ppEnabledLayerNames, enabledExtensionCount, ppEnabledExtensionNames, pEnabledFeatures -- `VkPhysicalDeviceFeatures` — all VkBool32 (can be zeroed for triangle) -- `VkPhysicalDeviceDynamicRenderingFeatures` — sType, pNext, dynamicRendering (VkBool32) — needed to enable dynamic rendering -- `VkSwapchainCreateInfoKHR` — sType, pNext, flags, surface, minImageCount, imageFormat, imageColorSpace, imageExtent, imageArrayLayers, imageUsage, imageSharingMode, queueFamilyIndexCount, pQueueFamilyIndices, preTransform, compositeAlpha, presentMode, clipped, oldSwapchain -- `VkImageViewCreateInfo` — sType, pNext, flags, image, viewType, format, components, subresourceRange -- `VkComponentMapping` — r, g, b, a (VkComponentSwizzle) -- `VkImageSubresourceRange` — aspectMask, baseMipLevel, levelCount, baseArrayLayer, layerCount -- `VkExtent2D` — width, height -- `VkExtent3D` — width, height, depth -- `VkOffset2D` — x, y -- `VkOffset3D` — x, y, z -- `VkRect2D` — offset, extent -- `VkViewport` — x, y, width, height, minDepth, maxDepth -- `VkShaderModuleCreateInfo` — sType, pNext, flags, codeSize, pCode -- `VkPipelineShaderStageCreateInfo` — sType, pNext, flags, stage, module, pName, pSpecializationInfo -- `VkPipelineVertexInputStateCreateInfo` — sType, pNext, flags, vertexBindingDescriptionCount, pVertexBindingDescriptions, vertexAttributeDescriptionCount, pVertexAttributeDescriptions -- `VkVertexInputBindingDescription` — binding, stride, inputRate -- `VkVertexInputAttributeDescription` — location, binding, format, offset -- `VkPipelineInputAssemblyStateCreateInfo` — sType, pNext, flags, topology, primitiveRestartEnable -- `VkPipelineViewportStateCreateInfo` — sType, pNext, flags, viewportCount, pViewports, scissorCount, pScissors -- `VkPipelineRasterizationStateCreateInfo` — sType, pNext, flags, depthClampEnable, rasterizerDiscardEnable, polygonMode, cullMode, frontFace, depthBiasEnable, depthBiasConstantFactor, depthBiasClamp, depthBiasSlopeFactor, lineWidth -- `VkPipelineMultisampleStateCreateInfo` — sType, pNext, flags, rasterizationSamples, sampleShadingEnable, minSampleShading, pSampleMask, alphaToCoverageEnable, alphaToOneEnable -- `VkPipelineColorBlendAttachmentState` — blendEnable, srcColorBlendFactor, dstColorBlendFactor, colorBlendOp, srcAlphaBlendFactor, dstAlphaBlendFactor, alphaBlendOp, colorWriteMask -- `VkPipelineColorBlendStateCreateInfo` — sType, pNext, flags, logicOpEnable, logicOp, attachmentCount, pAttachments, blendConstants[4] -- `VkPipelineDynamicStateCreateInfo` — sType, pNext, flags, dynamicStateCount, pDynamicStates -- `VkPipelineLayoutCreateInfo` — sType, pNext, flags, setLayoutCount, pSetLayouts, pushConstantRangeCount, pPushConstantRanges -- `VkGraphicsPipelineCreateInfo` — sType, pNext, flags, stageCount, pStages, pVertexInputState, pInputAssemblyState, pViewportState, pRasterizationState, pMultisampleState, pDepthStencilState, pColorBlendState, pDynamicState, layout, renderPass, subpass, basePipelineHandle, basePipelineIndex -- `VkCommandPoolCreateInfo` — sType, pNext, flags, queueFamilyIndex -- `VkCommandBufferAllocateInfo` — sType, pNext, commandPool, level, commandBufferCount -- `VkCommandBufferBeginInfo` — sType, pNext, flags, pInheritanceInfo -- `VkSemaphoreCreateInfo` — sType, pNext, flags -- `VkFenceCreateInfo` — sType, pNext, flags -- `VkBufferCreateInfo` — sType, pNext, flags, size, usage, sharingMode, queueFamilyIndexCount, pQueueFamilyIndices -- `VkMemoryAllocateInfo` — sType, pNext, allocationSize, memoryTypeIndex -- `VkMemoryRequirements` — size, alignment, memoryTypeBits -- `VkPhysicalDeviceMemoryProperties` — memoryTypeCount, memoryTypes[32], memoryHeapCount, memoryHeaps[16] -- `VkMemoryType` — propertyFlags, heapIndex -- `VkMemoryHeap` — size, flags -- `VkQueueFamilyProperties` — queueFlags, queueCount, timestampValidBits, minImageTransferGranularity -- `VkSurfaceCapabilitiesKHR` — minImageCount, maxImageCount, currentExtent, minImageExtent, maxImageExtent, maxImageArrayLayers, supportedTransforms, currentTransform, supportedCompositeAlpha, supportedUsageFlags -- `VkSurfaceFormatKHR` — format, colorSpace -- `VkPhysicalDeviceProperties` — apiVersion, driverVersion, vendorID, deviceID, deviceType, deviceName[256], ... -- `VkSubmitInfo` — sType, pNext, waitSemaphoreCount, pWaitSemaphores, pWaitDstStageMask, commandBufferCount, pCommandBuffers, signalSemaphoreCount, pSignalSemaphores -- `VkSubmitInfo2` — sType, pNext, flags, waitSemaphoreInfoCount, pWaitSemaphoreInfos, commandBufferInfoCount, pCommandBufferInfos, signalSemaphoreInfoCount, pSignalSemaphoreInfos (sync2) -- `VkSemaphoreSubmitInfo` — sType, pNext, semaphore, value, stageMask, deviceIndex (sync2) -- `VkCommandBufferSubmitInfo` — sType, pNext, commandBuffer, deviceMask (sync2) -- `VkPresentInfoKHR` — sType, pNext, waitSemaphoreCount, pWaitSemaphores, swapchainCount, pSwapchains, pImageIndices, pResults -- `VkClearValue` — union: VkClearColorValue color / VkClearDepthStencilValue depthStencil -- `VkClearColorValue` — union: float[4] / int[4] / uint[4] -- `VkClearDepthStencilValue` — depth, stencil -- `VkRenderingAttachmentInfo` — sType, pNext, imageView, imageLayout, resolveMode, resolveImageView, resolveImageLayout, loadOp, storeOp, clearValue -- `VkRenderingInfo` — sType, pNext, flags, renderArea, layerCount, viewMask, colorAttachmentCount, pColorAttachments, pDepthAttachment, pStencilAttachment -- `VkImageMemoryBarrier2` — sType, pNext, srcStageMask, srcAccessMask, dstStageMask, dstAccessMask, oldLayout, newLayout, srcQueueFamilyIndex, dstQueueFamilyIndex, image, subresourceRange -- `VkBufferMemoryBarrier2` — sType, pNext, srcStageMask, srcAccessMask, dstStageMask, dstAccessMask, srcQueueFamilyIndex, dstQueueFamilyIndex, buffer, offset, size -- `VkDependencyInfo` — sType, pNext, dependencyFlags, memoryBarrierCount, pMemoryBarriers, bufferMemoryBarrierCount, pBufferMemoryBarriers, imageMemoryBarrierCount, pImageMemoryBarriers -- `VkBufferCopy` — srcOffset, dstOffset, size -- `VkDebugUtilsMessengerCallbackDataEXT` — sType, pNext, messageId, pMessageIdName, messageSeverity, messageType, pMessage, queueLabelCount, pQueueLabels, cmdBufLabelCount, pCmdBufLabels, objectCount, pObjects -- `VkDebugUtilsObjectNameInfoEXT` — sType, pNext, objectType, objectHandle, pObjectName - -#### 1.5 `Vk.cs` -Function delegate types + loaded function pointers: - -**Instance-level functions** (loaded via `vkGetInstanceProcAddr`): -- `vkCreateInstance`, `vkDestroyInstance` -- `vkEnumeratePhysicalDevices`, `vkGetPhysicalDeviceProperties`, `vkGetPhysicalDeviceMemoryProperties` -- `vkGetPhysicalDeviceQueueFamilyProperties` -- `vkGetPhysicalDeviceSurfaceSupportKHR` -- `vkGetPhysicalDeviceSurfaceCapabilitiesKHR`, `vkGetPhysicalDeviceSurfaceFormatsKHR`, `vkGetPhysicalDeviceSurfacePresentModesKHR` -- `vkCreateDevice`, `vkDestroyDevice` -- `vkDestroySurfaceKHR` -- `vkCreateDebugUtilsMessengerEXT`, `vkDestroyDebugUtilsMessengerEXT` (extension — via getInstanceProcAddr) -- `vkGetDeviceProcAddr` - -**Device-level functions** (loaded via `vkGetDeviceProcAddr` for best performance): -- `vkGetDeviceQueue` -- `vkCreateSwapchainKHR`, `vkDestroySwapchainKHR`, `vkGetSwapchainImagesKHR` -- `vkCreateImageView`, `vkDestroyImageView` -- `vkCreateShaderModule`, `vkDestroyShaderModule` -- `vkCreatePipelineLayout`, `vkDestroyPipelineLayout` -- `vkCreateGraphicsPipelines`, `vkDestroyPipeline` -- `vkCreateCommandPool`, `vkDestroyCommandPool` -- `vkAllocateCommandBuffers`, `vkFreeCommandBuffers` -- `vkBeginCommandBuffer`, `vkEndCommandBuffer`, `vkResetCommandBuffer` -- `vkCreateSemaphore`, `vkDestroySemaphore` -- `vkCreateFence`, `vkDestroyFence`, `vkResetFences`, `vkWaitForFences`, `vkGetFenceStatus` -- `vkCreateBuffer`, `vkDestroyBuffer` -- `vkAllocateMemory`, `vkFreeMemory` -- `vkBindBufferMemory` -- `vkGetBufferMemoryRequirements` -- `vkMapMemory`, `vkUnmapMemory` -- `vkCmdBindPipeline` -- `vkCmdSetViewport`, `vkCmdSetScissor` -- `vkCmdBindVertexBuffers` -- `vkCmdDraw` -- `vkCmdBeginRendering`, `vkCmdEndRendering` (Vulkan 1.3 dynamic rendering) -- `vkCmdPipelineBarrier2` (sync2) -- `vkCmdCopyBuffer` -- `vkCmdBindIndexBuffer`, `vkCmdDrawIndexed` (for future) -- `vkAcquireNextImageKHR` -- `vkQueueSubmit2` (sync2) -- `vkQueuePresentKHR` -- `vkDeviceWaitIdle` -- `vkQueueWaitIdle` - -### Phase 2: Vulkan Context - -#### 2.1 `VulkanContext.cs` -- **CreateInstance:** - - `VkApplicationInfo` with `apiVersion = VK_API_VERSION_1_3` - - Instance extensions from SDL3: `SDL_GetVulkanInstanceExtensions()` - - Add `VK_EXT_debug_utils` in Debug - - Layers: `VK_LAYER_KHRONOS_validation` in Debug - - Chain `VkDebugUtilsMessengerCreateInfoEXT` in `pNext` for early validation - - Debug callback: prints `pMessage` to stderr/console - -- **PickPhysicalDevice:** - - Enumerate all physical devices - - Prefer `VK_PHYSICAL_DEVICE_TYPE_DISCRETE_GPU` - - Find queue family with `VK_QUEUE_GRAPHICS_BIT` + surface support (`vkGetPhysicalDeviceSurfaceSupportKHR`) - -- **CreateLogicalDevice:** - - Enable `VK_KHR_swapchain` device extension - - Chain `VkPhysicalDeviceDynamicRenderingFeatures` in `pNext` with `dynamicRendering = VK_TRUE` - - Single queue from selected family, priority 1.0 - -- **CreateSurface:** - - Call SDL3 `SDL_Vulkan_CreateSurface(window, instance, ...)` via Sdl3Window - - Store `VkSurfaceKHR` - -- **Debug Messenger:** - - `vkCreateDebugUtilsMessengerEXT` with callback - - Severity: Verbose | Warning | Error - - Type: General | Validation | Performance - -### Phase 3: Swapchain - -#### 3.1 `VulkanSwapchain.cs` -- **Query surface:** - - `vkGetPhysicalDeviceSurfaceCapabilitiesKHR` → min/max image count, current extent - - `vkGetPhysicalDeviceSurfaceFormatsKHR` → prefer `B8G8R8A8_UNORM` + `SrgbNonlinear`, fallback first format - - `vkGetPhysicalDeviceSurfacePresentModesKHR` → prefer `MAILBOX`, fallback `FIFO` (guaranteed) - -- **Create swapchain:** - - `minImageCount = max(minImageCount + 1, maxImageCount)` (clamped) - - `imageUsage = COLOR_ATTACHMENT_BIT | TRANSFER_DST_BIT` (for future screenshots) - - `preTransform = currentTransform` (no pre-rotation on desktop) - - `compositeAlpha = OPAQUE_BIT` - - `clipped = VK_TRUE` - - `oldSwapchain = VK_NULL_HANDLE` (on first create) - -- **Get swapchain images:** - - `vkGetSwapchainImagesKHR` → array of `VkImage` - - Create `VkImageView` for each (`TYPE_2D`, same format, `COLOR_BIT` aspect) - -- **Recreate:** - - `vkDeviceWaitIdle` - - Destroy old image views + swapchain - - Create new swapchain with `oldSwapchain` = old handle - - Create new image views - -### Phase 4: Pipeline - -#### 4.1 `VulkanPipeline.cs` -- **Shader modules:** - - Load `triangle.vert.spv` and `triangle.frag.spv` from embedded resources or filesystem - - `vkCreateShaderModule` for each - -- **Vertex input:** - - Binding 0: stride = sizeof(Vertex) = 36 bytes, `VERTEX_INPUT_RATE_VERTEX` - - Attribute 0: `R32G32B32_SFLOAT` @ offset 0 (Position, location 0) - - Attribute 1: `R32G32B32_SFLOAT` @ offset 12 (Color, location 1) - - Attribute 2: `R32G32B32_SFLOAT` @ offset 24 (Normal, location 2) - -- **Pipeline state:** - - Input assembly: `TRIANGLE_LIST` - - Viewport state: viewportCount=1, scissorCount=1 (dynamic values) - - Rasterization: `FILL`, cull `NONE`, `COUNTER_CLOCKWISE`, lineWidth=1.0 - - Multisample: `COUNT_1_BIT`, no sample shading - - Color blend: 1 attachment, blend disabled, write RGBA - - Dynamic state: `VIEWPORT`, `SCISSOR` - - Pipeline layout: no descriptor sets, no push constants (triangle only) - -- **Dynamic rendering integration:** - - `VkGraphicsPipelineCreateInfo::renderPass = VK_NULL_HANDLE` (Vulkan 1.3 dynamic rendering) - - Set `pNext` to `VkPipelineRenderingCreateInfo` with `colorAttachmentCount=1`, `pColorAttachmentFormats = {swapchainFormat}` - -### Phase 5: Frame Resources - -#### 5.1 `VulkanFrameResources.cs` -- **Constants:** - - `MAX_FRAMES_IN_FLIGHT = 2` - -- **Per-frame-in-flight resources** (indexed 0..MAX_FRAMES_IN_FLIGHT-1): - - `VkCommandBuffer` — primary, from shared command pool - - `VkFence` — signaled on submit, waited at frame start (created with `SIGNALED` flag) - - `VkSemaphore` — acquire semaphore (signaled by `vkAcquireNextImageKHR`) - -- **Per-swapchain-image resources** (indexed 0..swapchainImageCount-1): - - `VkSemaphore` — submit/render-finished semaphore (signaled by `vkQueueSubmit2`, waited by `vkQueuePresentKHR`) - - **CRITICAL:** These are indexed by swapchain image index, NOT frame-in-flight index. - This is the correct pattern from the Vulkan Guide (§swapchain_semaphore_reuse). - Waiting on the acquire semaphore/fence for a given image index guarantees the previous - present operation using that image has completed, making the submit semaphore safe to reuse. - -- **Command pool:** - - `vkCreateCommandPool` with `RESET_COMMAND_BUFFER_BIT` flag - - Allocate `MAX_FRAMES_IN_FLIGHT` primary command buffers - -### Phase 6: Vertex Buffer - -#### 6.1 `VulkanVertexBuffer.cs` -- **Staging buffer pattern (correct from start):** - 1. Create staging buffer: `usage = TRANSFER_SRC_BIT`, memory = `HOST_VISIBLE | HOST_COHERENT` - 2. `vkMapMemory` → `memcpy` vertex data → `vkUnmapMemory` - 3. Create vertex buffer: `usage = TRANSFER_DST_BIT | VERTEX_BUFFER_BIT`, memory = `DEVICE_LOCAL` - 4. Allocate + record one-time command buffer - 5. `vkCmdCopyBuffer(staging, vertex, size)` - 6. Submit + wait on fence - 7. Destroy staging buffer + free its memory + free one-time command buffer - -- **Triangle data:** - ``` - Vertex[3] = { - { Position: ( 0.0, -0.5, 0.0), Color: (1, 0, 0), Normal: (0, 0, 1) }, - { Position: ( 0.5, 0.5, 0.0), Color: (0, 1, 0), Normal: (0, 0, 1) }, - { Position: (-0.5, 0.5, 0.0), Color: (0, 0, 1), Normal: (0, 0, 1) }, - } - ``` - -- **Memory type selection:** - - `vkGetPhysicalDeviceMemoryProperties` → iterate `memoryTypes[]` - - Find type where `(memoryTypeBits >> i) & 1` and `propertyFlags` matches desired flags - - Helper: `FindMemoryType(memoryTypeBits, desiredFlags)` - -### Phase 7: Renderer - -#### 7.1 `VulkanRenderer.cs` (implements `IRenderer`) -- **Constructor:** - - Create swapchain, pipeline, frame resources, vertex buffer - - Store reference to `VulkanContext` (instance, device, queue, surface) - -- **Frame loop (`Render()` method):** - ``` - 1. vkWaitForFences(frameFences[frameIndex]) - 2. vkResetFences(frameFences[frameIndex]) - 3. vkAcquireNextImageKHR(swapchain, acquireSemaphores[frameIndex], imageIndex) - 4. vkResetCommandBuffer(commandBuffers[frameIndex]) - 5. vkBeginCommandBuffer(commandBuffers[frameIndex], ONE_TIME_SUBMIT) - 6. Image layout transition (sync2 barrier): - UNDEFINED → COLOR_ATTACHMENT_OPTIMAL - (srcStageMask: NONE, dstStageMask: COLOR_ATTACHMENT_OUTPUT) - 7. vkCmdBeginRendering(renderingInfo): - - colorAttachment: swapchainImageViews[imageIndex], COLOR_ATTACHMENT_OPTIMAL - - loadOp: CLEAR (black), storeOp: STORE - - renderArea: full extent - 8. vkCmdBindPipeline(GRAPHICS, pipeline) - 9. vkCmdSetViewport(0, 1, {0, 0, extent.width, extent.height, 0, 1}) - 10. vkCmdSetScissor(0, 1, {{0,0}, extent}) - 11. vkCmdBindVertexBuffers(0, 1, {vertexBuffer}, {0}) - 12. vkCmdDraw(3, 1, 0, 0) - 13. vkCmdEndRendering() - 14. Image layout transition (sync2 barrier): - COLOR_ATTACHMENT_OPTIMAL → PRESENT_SRC_KHR - (srcStageMask: COLOR_ATTACHMENT_OUTPUT, dstStageMask: ALL_GRAPHICS) - 15. vkEndCommandBuffer() - 16. vkQueueSubmit2(queue, submitInfo2): - - wait: acquireSemaphores[frameIndex] @ COLOR_ATTACHMENT_OUTPUT - - commandBuffer: commandBuffers[frameIndex] - - signal: submitSemaphores[imageIndex] - - fence: frameFences[frameIndex] - 17. vkQueuePresentKHR(presentInfo): - - wait: submitSemaphores[imageIndex] - - swapchain, imageIndex - 18. frameIndex = (frameIndex + 1) % MAX_FRAMES_IN_FLIGHT - ``` - -- **Resize handling:** - - If `vkAcquireNextImageKHR` returns `ERROR_OUT_OF_DATE_KHR` or `SuboptimalKHR`: - - `vkDeviceWaitIdle` - - Recreate swapchain - - Continue frame - -- **Dispose:** - - `vkDeviceWaitIdle` - - Destroy vertex buffer + memory - - Destroy semaphores (acquire + submit), fences - - Destroy command pool - - Destroy pipeline, pipeline layout, shader modules - - Destroy swapchain + image views - - Destroy debug messenger - - Destroy device, surface, instance - -#### 7.2 `VulkanRenderContext.cs` (implements `IRenderContext`) -- Exposes `Window` (from Sdl3Window) -- `CreateRenderer()` → returns `VulkanRenderer` -- `Resize()` → triggers swapchain recreation -- `Dispose()` → destroys context - -#### 7.3 `VulkanBackendRegistrar.cs` -- Static constructor registers `"vulkan"` in `RenderBackendFactory` -- Factory creates `VulkanRenderContext` with `Sdl3Window` - -### Phase 8: Shaders - -#### 8.1 `Shaders/triangle.vert` -```glsl -#version 450 - -layout(location = 0) in vec3 inPosition; -layout(location = 1) in vec3 inColor; -layout(location = 2) in vec3 inNormal; - -layout(location = 0) out vec3 fragColor; - -void main() { - gl_Position = vec4(inPosition, 1.0); - fragColor = inColor; -} -``` - -#### 8.2 `Shaders/triangle.frag` -```glsl -#version 450 - -layout(location = 0) in vec3 fragColor; -layout(location = 0) out vec4 outColor; - -void main() { - outColor = vec4(fragColor, 1.0); -} -``` - -#### 8.3 Compilation -```bash -glslangValidator -V triangle.vert -o triangle.vert.spv -glslangValidator -V triangle.frag -o triangle.frag.spv -``` -- Embed `.spv` files as embedded resources in csproj, or copy to output directory -- Load at runtime via `Assembly.GetManifestResourceStream()` or `File.ReadAllBytes()` - -### Phase 9: App Integration - -- Fix `CortexEngine.App.csproj`: - - Remove deleted project references - - Add `Engine.Graphics` + `Engine.Graphics.Vulkan` -- Fix `Program.cs`: - - Simplify to triangle-only rendering - - `RenderBackendFactory.Create("vulkan", 1280, 720, validation: true)` - - Main loop: poll events → render → present - - Keep: Sdl3Window, basic event handling - - Remove: ECS scene, physics, AI, camera tour (add back later) - -### Phase 10: Fix Tests - -- Update `Engine.Tests.csproj` — reference restored `Engine.Graphics` -- Tests referencing Engine.Graphics: ObjLoaderTests, RenderBackendFactoryTests, - SceneSerializerTests, MeshMathAndProceduralTests -- All tests should pass after Engine.Graphics is restored - ---- - -## File Layout - -``` -src/ -├── Engine.Core/ (exists, unchanged) -├── Engine.Graphics/ (exists, restored minimal interfaces) -│ ├── Engine.Graphics.csproj -│ ├── IRenderContext.cs -│ ├── IRenderer.cs -│ ├── IScreenshotProvider.cs -│ ├── RenderBackendFactory.cs -│ ├── MeshMath.cs -│ ├── ProceduralMesh.cs -│ ├── SceneSerializer.cs -│ └── Loaders/ -│ └── ObjLoader.cs -├── Engine.Graphics.Vulkan/ (new — pure P/Invoke, Vulkan 1.3) -│ ├── Engine.Graphics.Vulkan.csproj -│ ├── VulkanNative.cs — library loading, vkGetInstanceProcAddr -│ ├── VulkanHandles.cs — opaque pointer types -│ ├── VulkanEnums.cs — all Vulkan enums/flags -│ ├── VulkanStructs.cs — all Vulkan structs (LayoutKind.Sequential) -│ ├── Vk.cs — function delegates + loaded pointers -│ ├── VulkanContext.cs — instance, device, queue, surface, debug -│ ├── VulkanSwapchain.cs — swapchain, image views, recreate -│ ├── VulkanPipeline.cs — shader modules, pipeline layout, graphics pipeline -│ ├── VulkanFrameResources.cs — command buffers, fences, semaphores (correct indexing) -│ ├── VulkanVertexBuffer.cs — staging buffer → device-local vertex buffer -│ ├── VulkanRenderer.cs — IRenderer: frame loop with dynamic rendering -│ ├── VulkanRenderContext.cs — IRenderContext implementation -│ ├── VulkanBackendRegistrar.cs— registration in RenderBackendFactory -│ └── Shaders/ -│ ├── triangle.vert -│ ├── triangle.frag -│ ├── triangle.vert.spv -│ └── triangle.frag.spv -├── Engine.Physics/ (exists, unchanged) -├── Engine.AI/ (exists, unchanged) -└── CortexEngine.App/ (fix references, simplify to triangle) -``` - ---- - -## csproj: Engine.Graphics.Vulkan - -```xml - - - net9.0 - true - true - - - - - - - - - -``` - -No NuGet packages for Vulkan. Pure P/Invoke. - ---- - -## Cross-Platform Notes - -- **Library name:** `vulkan-1.dll` (Windows) vs `libvulkan.so.1` (Linux) — handled in `VulkanNative.cs` -- **Surface creation:** SDL3 abstracts platform differences (`SDL_Vulkan_CreateSurface`) -- **SPIR-V:** Binary format, identical on all platforms -- **.NET 9:** `NativeLibrary.Load()` for dynamic resolution - ---- - -## Key Technical Details - -### Semaphore Indexing (CRITICAL) - -``` - Indexed by frame-in-flight (0..1) Indexed by swapchain image (0..N-1) - ───────────────────────────────── ────────────────────────────────── -Acquire semaphore ✓ -Command buffer ✓ -Frame fence ✓ -Submit semaphore ✓ -``` - -Rationale: `vkQueuePresentKHR` cannot signal a fence/semaphore. The only way to know -a submit semaphore is safe to reuse is to acquire the same swapchain image index again -(which guarantees the previous present using that image has completed). -Indexing submit semaphores by frame-in-flight is a common bug that violates the spec. - -### Dynamic Rendering (Vulkan 1.3) - -No `VkRenderPass` or `VkFramebuffer` objects needed: -```csharp -// Instead of vkCmdBeginRenderPass: -VkRenderingAttachmentInfo colorAttachment = new() { - sType = VK_STRUCTURE_TYPE_RENDERING_ATTACHMENT_INFO, - imageView = swapchainImageViews[imageIndex], - imageLayout = COLOR_ATTACHMENT_OPTIMAL, - loadOp = CLEAR, - storeOp = STORE, - clearValue = new() { color = { 0, 0, 0, 1 } } -}; - -VkRenderingInfo renderingInfo = new() { - sType = VK_STRUCTURE_TYPE_RENDERING_INFO, - renderArea = { {0,0}, extent }, - layerCount = 1, - colorAttachmentCount = 1, - pColorAttachments = &colorAttachment -}; - -vkCmdBeginRendering(commandBuffer, &renderingInfo); -// draw commands... -vkCmdEndRendering(commandBuffer); -``` - -Pipeline must include `VkPipelineRenderingCreateInfo` in `pNext`: -```csharp -VkPipelineRenderingCreateInfo renderingInfo = new() { - sType = VK_STRUCTURE_TYPE_PIPELINE_RENDERING_CREATE_INFO, - colorAttachmentCount = 1, - pColorAttachmentFormats = &swapchainFormat -}; -// Chain in VkGraphicsPipelineCreateInfo.pNext -``` - -### Sync2 Image Layout Transitions - -Using `vkCmdPipelineBarrier2` with `VkImageMemoryBarrier2`: -```csharp -// UNDEFINED → COLOR_ATTACHMENT_OPTIMAL (before rendering) -VkImageMemoryBarrier2 toColor = new() { - sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, - srcStageMask = PIPELINE_STAGE_2_NONE, - srcAccessMask = ACCESS_2_NONE, - dstStageMask = PIPELINE_STAGE_2_COLOR_ATTACHMENT_OUTPUT, - dstAccessMask = ACCESS_2_COLOR_ATTACHMENT_WRITE, - oldLayout = UNDEFINED, - newLayout = COLOR_ATTACHMENT_OPTIMAL, - image = swapchainImages[imageIndex], - subresourceRange = { COLOR_BIT, 0, 1, 0, 1 } -}; - -// COLOR_ATTACHMENT_OPTIMAL → PRESENT_SRC_KHR (after rendering) -VkImageMemoryBarrier2 toPresent = new() { - sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, - srcStageMask = PIPELINE_STAGE_2_COLOR_ATTACHMENT_OUTPUT, - srcAccessMask = ACCESS_2_COLOR_ATTACHMENT_WRITE, - dstStageMask = PIPELINE_STAGE_2_ALL_GRAPHICS, - dstAccessMask = ACCESS_2_NONE, - oldLayout = COLOR_ATTACHMENT_OPTIMAL, - newLayout = PRESENT_SRC_KHR, - image = swapchainImages[imageIndex], - subresourceRange = { COLOR_BIT, 0, 1, 0, 1 } -}; - -VkDependencyInfo depInfo = new() { - sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, - imageMemoryBarrierCount = 1, - pImageMemoryBarriers = &barrier -}; -vkCmdPipelineBarrier2(commandBuffer, &depInfo); -``` - -### Queue Submit (Sync2) - -Using `vkQueueSubmit2` with `VkSubmitInfo2`: -```csharp -VkSemaphoreSubmitInfo waitInfo = new() { - sType = VK_STRUCTURE_TYPE_SEMAPHORE_SUBMIT_INFO, - semaphore = acquireSemaphores[frameIndex], - stageMask = PIPELINE_STAGE_2_COLOR_ATTACHMENT_OUTPUT -}; - -VkCommandBufferSubmitInfo cmdInfo = new() { - sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, - commandBuffer = commandBuffers[frameIndex] -}; - -VkSemaphoreSubmitInfo signalInfo = new() { - sType = VK_STRUCTURE_TYPE_SEMAPHORE_SUBMIT_INFO, - semaphore = submitSemaphores[imageIndex], - stageMask = PIPELINE_STAGE_2_ALL_GRAPHICS -}; - -VkSubmitInfo2 submitInfo = new() { - sType = VK_STRUCTURE_TYPE_SUBMIT_INFO_2, - waitSemaphoreInfoCount = 1, - pWaitSemaphoreInfos = &waitInfo, - commandBufferInfoCount = 1, - pCommandBufferInfos = &cmdInfo, - signalSemaphoreInfoCount = 1, - pSignalSemaphoreInfos = &signalInfo -}; - -vkQueueSubmit2(queue, 1, &submitInfo, frameFences[frameIndex]); -``` - -### Vertex Layout - -``` -Vertex struct (9 floats, 36 bytes): - Position: vec3 (offset 0, format R32G32B32_SFLOAT, location 0) - Color: vec3 (offset 12, format R32G32B32_SFLOAT, location 1) - Normal: vec3 (offset 24, format R32G32B32_SFLOAT, location 2) -``` - -### Validation Layers - -```csharp -string[] layers = enableValidation - ? new[] { "VK_LAYER_KHRONOS_validation" } - : Array.Empty(); - -string[] instanceExtensions = enableValidation - ? [.. sdlExtensions, "VK_EXT_debug_utils"] - : sdlExtensions; -``` - -Debug callback (C#): -```csharp -static uint DebugCallback( - nint instance, uint messageSeverity, uint messageTypes, - nint pCallbackData, nint pUserData) -{ - var data = Marshal.PtrToStructure(pCallbackData); - Console.Error.WriteLine($"[Vulkan] {data.pMessage}"); - return 0; // VK_FALSE — don't abort -} -``` - ---- - -## Future Phases (Not in This Plan) - -- **Phase 11:** ImGui integration (ImGui.NET + Vulkan backend) -- **Phase 12:** Mesh rendering (OBJ loading, index buffers, descriptor sets, UBO for camera) -- **Phase 13:** PBR shading (Fresnel, ACES tonemap, gamma correction, directional + point lights) -- **Phase 14:** Shadow mapping (depth-only render pass from light POV, PCF sampling) -- **Phase 15:** Screenshot capture (copy swapchain image to staging buffer → PNG) -- **Phase 16:** VMA (Vulkan Memory Allocator) for sub-allocation -- **Phase 17:** Multi-threaded command buffer recording +## Status: COMPLETE + +All phases implemented. See `CORTEX_ENGINE_ARCHITECTURE.md` for current architecture. + +## Completed Phases + +1. ✅ Vulkan P/Invoke foundation (Vulkan 1.3, dynamic rendering, sync2) +2. ✅ Push constants (model matrix) +3. ✅ UBO + descriptor sets (SceneUBO with multi-light data) +4. ✅ Index buffer + OBJ loading +5. ✅ Depth buffer (D32_SFLOAT) +6. ✅ Face normal computation +7. ✅ FreeFlyCameraController (WASD + mouse look) +8. ✅ Projection matrix fix (Y-flip + no row_major) +9. ✅ ECS scene (mesh cache, per-entity model matrix) +10. ✅ PBR shading (Cook-Torrance, ACES tonemapping) +11. ✅ Jolt physics (gravity, collision, floor) +12. ✅ ImGui debug overlay +13. ✅ AI/MCP (7 tools, HTTP SSE) +14. ✅ Cubemap shadow mapping (omnidirectional, 6 faces per light) +15. ✅ Multi-light system (cubemap array, per-light shadows) +16. ✅ Soft shadows (16-tap Poisson disk PCF) +17. ✅ Shadow parameters ImGui panel +18. ✅ Video recording (FFmpeg pipe) +19. ✅ Adjustable ambient lighting +20. ✅ Physics pause/resume + scene reset +21. ✅ 227 xUnit tests diff --git a/imgui.ini b/imgui.ini deleted file mode 100644 index bce0f71..0000000 --- a/imgui.ini +++ /dev/null @@ -1,40 +0,0 @@ -[Window][Debug##Default] -Pos=60,60 -Size=400,400 -Collapsed=0 - -[Window][Debug] -Pos=10,10 -Size=216,107 -Collapsed=0 - -[Window][Hierarchy] -Pos=353,299 -Size=250,398 -Collapsed=0 - -[Window][Inspector] -Pos=112,297 -Size=300,425 -Collapsed=0 - -[Window][Cortex Engine Debug] -Pos=1,1 -Size=226,196 -Collapsed=0 - -[Window][Shadow Parameters] -Pos=772,456 -Size=507,263 -Collapsed=0 - -[Window][Shadow & Light Parameters] -Pos=22,191 -Size=223,506 -Collapsed=0 - -[Window][Video Recording] -Pos=986,3 -Size=284,54 -Collapsed=0 - diff --git a/scripts/run.sh b/scripts/run.sh index 8c3bc6c..1e6abd4 100755 --- a/scripts/run.sh +++ b/scripts/run.sh @@ -2,17 +2,14 @@ # Run the Cortex Engine. # Examples: # ./scripts/run.sh # run with defaults -# ./scripts/run.sh --mcp-port 5000 # run with MCP HTTP server -# ./scripts/run.sh --camera-tour # capture screenshots and exit -# ./scripts/run.sh --mcp-stdio # run headless stdio MCP server +# ./scripts/run.sh -- --mcp-port 5000 # run with MCP HTTP server +# ./scripts/run.sh -- --mcp-port 0 # run without MCP ENGINE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" export DOTNET_ROOT="${DOTNET_ROOT:-$HOME/.dotnet}" export PATH="$DOTNET_ROOT:$PATH" -# Auto-detect Wayland. SDL3 bundled with ppy.SDL3-CS supports both Wayland and X11; -# let it choose based on available display servers. Do NOT force DISPLAY here. unset SDL_VIDEODRIVER 2>/dev/null cd "$ENGINE_DIR"