# CORTEX ENGINE — Technical Architecture Manifesto
**Project**: AI-Native, Multiplatform 3D Game Engine
**Language**: C# (.NET 9)
**Status**: LOCKED — All dependencies verified as of June 2026
**Created**: 2026-06-16
---
## 1. EXECUTIVE SUMMARY
Cortex Engine is a 3D game engine built from scratch to provide a Unity-like development experience (GameObject/Component paradigm, Inspector, Hierarchy, Scene View) while being deeply integrated with Multimodal Large Language Models (MMLMs). The engine allows an AI to:
- **See** the engine state via rendered-frame screenshots, virtual cameras, semantic segmentation maps, and profiler screenshots.
- **Read** the complete ECS world state through native JSON serialization.
- **Modify** the running engine via declarative JSON commands and, in Development Mode, via hot-reloaded C# scripts.
The architecture prioritizes **production maturity** over experimental technologies: a Render HAL with a Raylib-cs default backend and an optional Vulkan (Silk.NET.Vulkan) backend, Flecs.NET (C# bindings for the C-based Flecs ECS), SDL3-cs (ppy.SDL3-CS), and Hexa.NET.ImGui with a native backend.
---
## 2. EVOLUTION OF ARCHITECTURE DECISIONS
### 2.1 Initial Discussion
The project began with an exploration of NVIDIA's open-source physics and graphics ecosystems (PhysX 5, Newton, Warp, Falcor, Flow) and modern middleware for building a custom engine from scratch. Initial candidates included:
- **Graphics**: WebGPU (wgpu), Diligent Engine, BGFX, NVIDIA Falcor
- **Physics**: Jolt Physics, NVIDIA PhysX 5, Box2D v3
- **Windowing**: SDL3, GLFW
- **ECS**: Flecs, EnTT, Arch
- **UI**: Dear ImGui
### 2.2 First Iteration
The first proposed stack was:
- C# (.NET 9) + NativeAOT
- Silk.NET + wgpu-native (WebGPU)
- SDL3 (via Silk.NET)
- Flecs ECS
- Dear ImGui with custom WebGPU backend
- Jolt Physics
- Roslyn Compiler API for AI hot-reload
### 2.3 Identified Risks from Internet Research
Research revealed critical issues with the first iteration:
1. **NativeAOT + Roslyn conflict**: NativeAOT explicitly does not support `Assembly.LoadFile()` or `System.Reflection.Emit`. Dynamic C# compilation cannot run inside a NativeAOT binary. This is confirmed by Microsoft Learn documentation.
2. **Silk.NET WebGPU bindings**: The maintainers stated that the official WebGPU examples in Silk.NET are "very bad" and "smoke tests" — not production-ready.
3. **Custom WebGPU ImGui backend**: Would require writing ~400 lines of custom rendering code.
4. **Flecs vs Friflo tradeoff**: Friflo.Engine.ECS is pure managed and faster, but Flecs has mature native C-reflection and built-in JSON serialization that works identically in NativeAOT.
### 2.4 Final Locked Stack
The final stack was chosen to eliminate experimental dependencies and maximize production maturity:
- **C# (.NET 9) with dual-runtime strategy**: JIT for development (Roslyn hot-reload), NativeAOT for release.
- **SDL3-cs**: `ppy.SDL3-CS` — direct, zero-overhead P/Invoke bindings maintained by the osu! team.
- **Render HAL**: `Engine.Graphics` abstraction with pluggable backends.
- **Raylib-cs**: `Raylib-cs` 8.0.0 — default, simple OpenGL-based backend for rapid iteration and screenshot capture.
- **Vulkan**: `Silk.NET.Vulkan` 2.21.0 — optional high-performance backend retained as a reference implementation.
- **MoltenVK**: For macOS/iOS compatibility when using the Vulkan backend.
- **Flecs.NET**: `Flecs.NET.Release` — C# bindings for Flecs with NativeAOT static-link support.
- **ImGui**: `Hexa.NET.ImGui` — ships pre-built SDL3 + Vulkan native backends.
- **Jolt Physics**: `JoltPhysicsSharp` — C# bindings for Jolt Physics, .NET 9/10.
- **Roslyn Compiler API**: For Dev-Mode AI hot-reload.
- **SixLabors.ImageSharp**: For JPEG encoding of diagnostic textures.
---
## 3. TECHNOLOGICAL STACK (Zero-Contradiction Core)
### 3.1 Language & Runtime
**C# (.NET 9)**
The engine uses a **dual-runtime strategy**:
| Mode | Runtime | AI Scripting | Use Case |
|------|---------|--------------|----------|
| **Development** | .NET 9 JIT | ✅ Roslyn Compiler API + `AssemblyLoadContext` hot-reload | Editor, AI co-development, rapid iteration |
| **Release** | .NET 9 NativeAOT | ❌ JSON commands only | Shipped PC/Mobile/Console/WASM builds |
**Why this is not a contradiction:**
NativeAOT cannot JIT new code or load assemblies dynamically. Therefore, the engine builds in two configurations:
```xml
DEV_MODERELEASE_AOTtrue
```
All Roslyn and `AssemblyLoadContext` code is wrapped in `#if DEV_MODE`.
### 3.2 System Layer
**SDL3 via `ppy.SDL3-CS`**
- NuGet: `ppy.SDL3-CS` 2026.520.0
- Direct P/Invoke bindings, zero overhead
- Cross-platform: Windows, Linux, macOS, iOS, Android
- Handles: window creation, input (keyboard, mouse, touch, gamepad, accelerometer), audio, events
- Used by the osu! framework (2K+ stars), battle-tested
### 3.3 Graphics HAL
The graphics layer is split into a backend-agnostic **Render HAL** (`Engine.Graphics`) and concrete backend implementations.
**Core abstraction (`Engine.Graphics`)**
- `IRenderContext` — backend lifetime, resize, and surface handling.
- `IRenderer` — renders the ECS world and exposes screenshot capture.
- `RenderBackendFactory` — a registry/factory pattern; backend assemblies register themselves.
- The app depends only on these interfaces.
**Default backend: Raylib-cs**
- NuGet: `Raylib-cs` 8.0.0
- Simple, mature OpenGL-based renderer
- Handles window creation, mesh upload, 3D camera, and PNG screenshots internally
- Owns its GLFW window and input via `RaylibWindow` + `RaylibInputState` (no SDL3 dependency)
**Optional backend: Vulkan via `Silk.NET.Vulkan` — DEFERRED**
- NuGet: `Silk.NET.Vulkan` 2.21.0 and `Silk.NET.Vulkan.Extensions.KHR` 2.21.0
- The Vulkan backend compiles and implements the same `IRenderContext` / `IRenderer` HAL interfaces
- Uses `Sdl3Window` internally for Vulkan surface creation (`SDL_Vulkan_CreateSurface`)
- **Status: deferred to long-term backlog.** The backend is kept compilable and architecturally
integrated (via `IWindow`, `IRenderContext`), but is not actively tested or maintained.
The Raylib backend is the primary render path for all current development.
- **Reintegration checklist** (when picked up):
1. Test `VulkanRenderContext` with the new `IWindow`-based factory signature
2. Verify `SDL_Vulkan_CreateSurface` works through `IWindow.Handle`
3. Port improved shading (Fresnel, ACES, gamma, hemisphere ambient) to Vulkan GLSL shaders
4. Verify custom mesh upload (spheres, grids) works via Vulkan vertex/index buffers
5. Test screenshot capture via `ScreenshotCapture` with the new frame-deferral logic
**Why a HAL + Raylib default?**
- Drastically reduces the code the app, AI commands, and camera tools depend on
- Raylib-cs provides a fast, stable path for screenshots, 3D drawing, and windowing without custom shader/pipeline work
- Vulkan remains available as a high-performance, compute-capable backend for future vision pipelines
**macOS/iOS path:**
- When using the Vulkan backend: MoltenVK 1.4 supports Vulkan 1.4 on macOS, iOS, tvOS, visionOS
- `VK_KHR_portability_subset` and `VK_KHR_portability_enumeration` must be enabled
- Loader and MoltenVK libraries must be bundled with the application
- KosmicKrisp (via Mesa 3D) is an emerging alternative for Apple Silicon desktops
### 3.4 Data Architecture
**Flecs.NET**
- NuGet: `Flecs.NET.Debug` (Debug) / `Flecs.NET.Release` (Release) 4.0.4-build.546
- High-level C# wrapper over the C-based Flecs ECS
- Supports .NET Standard 2.1, .NET 5/6/7/8
- NativeAOT-compatible via static linking: `true`
- Native C-reflection: `ecs_world_to_json()` serializes the entire ECS world
- Used in AAA engines
**Why Flecs.NET over pure managed ECS (Friflo, Arch):**
- Native C-reflection works without C# `System.Reflection` — essential for NativeAOT
- Built-in JSON serialization of the entire world
- Production maturity and industry usage
- Component registration across shared libraries/DLLs
- Automatic archetype management and lockless scheduler
### 3.5 Editor UI
**Hexa.NET.ImGui**
- NuGet: `Hexa.NET.ImGui` (and related backend packages)
- Alternative to ImGui.NET that ships pre-built native backends
- Includes SDL3 + Vulkan backend combinations
- MIT licensed
- Higher performance, reduced startup time
- Eliminates the need to write a custom Vulkan renderer for ImGui
**Why Hexa.NET.ImGui over ImGui.NET:**
- `ImGui.NET` does not ship a C# Vulkan renderer out of the box
- The official Vulkan backend is `imgui_impl_vulkan.cpp` (C++), which must be manually compiled and P/Invoked
- Hexa.NET.ImGui bundles the C++ backends as native libraries with C# bindings
- This is the fastest path to a production-ready editor UI
### 3.6 Physics
**JoltPhysicsSharp**
- NuGet: `JoltPhysicsSharp` 2.21.0
- .NET 9/10 bindings for Jolt Physics
- Cross-platform via `joltc` C wrapper
- Used in Horizon Forbidden West and Death Stranding 2
- Integrated into Godot 4.4
**Note:** Physics module is not part of the foundational MVP but is included in the final project layout.
### 3.7 Diagnostic Encoding
**SixLabors.ImageSharp**
- Pure managed JPEG/PNG encoder
- NativeAOT-compatible
- No native dependencies
- Used to compress Vulkan-rendered RGBA textures into JPEG for MMLM vision input
---
## 4. DATA STRUCTURE & UNITY-LIKE ABSTRACTION
### 4.1 Core Principle
The engine exposes a Unity-like API surface (`GameObject`, `AddComponent`, `GetComponent`) while internally storing all data in Flecs components. The `GameObject` wrapper is **never** a place for state.
### 4.2 GameObject Facade
```csharp
public readonly struct GameObject
{
public readonly Entity Entity;
public readonly World World;
public GameObject(World world, Entity entity)
{
World = world;
Entity = entity;
}
public void AddComponent(T component) where T : unmanaged
{
Entity.Set(component);
}
public ref T GetComponent() where T : unmanaged
{
return ref Entity.GetMut();
}
public bool HasComponent() where T : unmanaged
{
return Entity.Has();
}
public void RemoveComponent() where T : unmanaged
{
Entity.Remove();
}
}
```
### 4.3 Component Definitions
Components are plain C# structs registered with the Flecs type system:
```csharp
public struct Transform : IComponent
{
public Vector3 Position;
public Quaternion Rotation;
public Vector3 Scale;
}
public struct MeshRef : IComponent
{
public ulong MeshId;
}
public struct Camera : IComponent
{
public Vector3 Position;
public Vector3 Target;
public Vector3 Up;
public float FieldOfView;
public float AspectRatio;
public float NearPlane;
public float FarPlane;
}
public struct Material : IComponent
{
public Vector3 Albedo;
public float Roughness;
public float Metallic;
public string? TexturePath;
}
public struct Light : IComponent
{
public Vector3 Direction;
public Vector3 Color;
public float Intensity;
}
public struct SemanticClass : IComponent
{
public byte ClassId; // 0=environment, 1=enemy, 2=player, 3=interactive, 4=trigger
}
```
### 4.4 AI Hot-Reloading (Dev Mode)
When the AI generates a C# script, the engine:
1. Receives the script string via the `AiGateway`.
2. Runs a pre-validation pass (syntax check, banned namespace check, unsafe code check).
3. Feeds the script to `Microsoft.CodeAnalysis.CSharp` (Roslyn).
4. Emits the compiled assembly into a `MemoryStream`.
5. Loads the assembly into a **dedicated** `AssemblyLoadContext`.
6. Extracts systems marked with `[Slot("name")]` attribute.
7. Calls `SystemSlotRegistry.HotSwap()` to replace the old system with the new one.
8. Migrates entities using the old component types to the new types.
9. Unloads the old `AssemblyLoadContext`.
**Important caveat:** Flecs stores component type metadata in its native C memory. If old C# types are still referenced by Flecs, the old `AssemblyLoadContext` cannot be fully unloaded. The migration step must remove old components and re-add them as the new types.
### 4.5 Rendering & Shading
The renderer uses a simple forward-lit pipeline that is implemented by each backend behind the HAL:
- **Vertex format**: position, color, normal.
- **Per-entity**: Mesh + Transform + optional Material.
- **Per-frame constants**: camera position, up to 4 directional lights, ambient color.
- **Per-entity constants**: MVP matrix, material albedo/roughness/metallic, texture use flag.
- **Lighting model**: multiple directional lights with ambient + diffuse + Blinn-Phong specular.
- **Material**: `Material.Albedo` tints vertex color, `Roughness` and `Metallic` control specular falloff and intensity; an optional `TexturePath` enables albedo texture sampling.
- **Vulkan backend**: uses a uniform buffer (descriptor set 0) and push constants; textures are Vulkan images with a combined image sampler (descriptor set 1).
- **Raylib backend**: uses a custom GLSL shader with `materialColor`, `useTexture`, `roughness`, `metallic`, and light arrays. Textures are loaded via `Raylib.LoadTexture` and UVs use world-space XZ.
- **UV mapping**: meshes use world-space XZ as a simple UV mapping for both backends.
### 4.6 SystemSlotRegistry
```csharp
public class SystemSlotRegistry
{
private readonly Dictionary _slots = new();
private readonly World _world;
public void Register(string slotName, Entity systemEntity)
{
_slots[slotName] = systemEntity;
}
public void HotSwap(string slotName, Entity newSystemEntity)
{
if (_slots.TryGetValue(slotName, out var oldSystem))
{
oldSystem.Destruct(); // Flecs system destructor
}
_slots[slotName] = newSystemEntity;
}
public IEnumerable> EnumerateSlots()
{
return _slots;
}
}
```
---
## 5. MMLM SELF-DIAGNOSTIC CONTEXT LOOP
### 5.1 Purpose
The `DiagnosticsManager` captures a multimodal **Diagnostic Payload** that gives the AI full context:
- **Visual context** — what the engine is currently rendering
- **Structural context** — the exact ECS world state
- **Code context** — current scripts, logs, and stack traces
### 5.2 Diagnostic Payload Structure
```csharp
public class DiagnosticPayload
{
public byte[] VisualJpeg { get; set; } // Semantic scene screenshot
public byte[] ProfilerJpeg { get; set; } // ImGui performance graph
public string WorldJson { get; set; } // Flecs ECS state
public string SystemGraphSvg { get; set; } // System pipeline diagram
public string ConsoleLogs { get; set; } // Recent log tail
public string StackTrace { get; set; } // Exception trace if any
public Dictionary SourceFiles { get; set; } // Active scripts
}
```
### 5.3 Capture Flow
```
[1] TRIGGER
│
├─ User clicks "AI Inspect" in the editor
├─ Unhandled exception occurs
└─ Automatic capture on frame-time spike
│
[2] CAPTURE
│
├─ [2a] Visual Layer (Vulkan)
│ ├─ Allocate off-screen VkImage (R8G8B8A8_UNORM)
│ ├─ Encode semantic render pass using SemanticClass component
│ ├─ Use Vulkan dynamic rendering (vkCmdBeginRendering / vkCmdEndRendering)
│ ├─ Copy image to host-visible staging buffer (vkCmdCopyImageToBuffer)
│ ├─ Map memory → Span RGBA
│ └─ Encode to JPEG via ImageSharp
│
├─ [2b] Visual Layer (Profiler)
│ ├─ Render ImGui profiler graph to the same RTT pipeline
│ └─ Encode to JPEG
│
├─ [2c] Structural Layer (Flecs)
│ ├─ Call ecs_world_to_json(world, ¶ms)
│ ├─ Marshal native UTF-8 pointer to managed string
│ └─ Optionally filter by camera frustum
│
├─ [2d] Structural Layer (System Graph)
│ ├─ Enumerate SystemSlotRegistry
│ └─ Generate SVG dependency graph
│
└─ [2e] Code & Log Layer
├─ Read circular log buffer
├─ Capture exception stack trace if present
└─ Read active scripts from /projects/
│
[3] PACK
│
└─ JSON envelope with base64-encoded JPEGs:
{
"visual": "",
"profiler": "",
"world": { ... flecs json ... },
"systems": "