feat: AI screenshot capture for visual analysis
- Add ScreenshotCapture class to Engine.Graphics using Vulkan image readback. - Integrate screenshot capture into MeshRenderer; request triggers readback on next frame. - Add SixLabors.ImageSharp 3.1.11 for PNG encoding (patched for CVE-2025-54575). - Add capture_screenshot AI command and MCP tool. - Wire screenshot request into Program.cs with demo command. - Expose swapchain surface format and image accessor for readback. - Add Screenshots/ to .gitignore. - Update CORTEX_ENGINE_ARCHITECTURE.md with MCP, Silk.NET.Vulkan, ImageSharp, and screenshot capture.
This commit is contained in:
@@ -15,6 +15,9 @@ obj/
|
|||||||
.DS_Store
|
.DS_Store
|
||||||
Thumbs.db
|
Thumbs.db
|
||||||
|
|
||||||
|
# Generated screenshots
|
||||||
|
Screenshots/
|
||||||
|
|
||||||
# dotnet
|
# dotnet
|
||||||
*.dll
|
*.dll
|
||||||
*.exe
|
*.exe
|
||||||
|
|||||||
@@ -11,11 +11,11 @@
|
|||||||
|
|
||||||
Cortex Engine is a 3D game engine built from scratch to provide a Unity-like development experience (GameObject/Component paradigm, Inspector, Hierarchy, Scene View) while being deeply integrated with Multimodal Large Language Models (MMLMs). The engine allows an AI to:
|
Cortex Engine is a 3D game engine built from scratch to provide a Unity-like development experience (GameObject/Component paradigm, Inspector, Hierarchy, Scene View) while being deeply integrated with Multimodal Large Language Models (MMLMs). The engine allows an AI to:
|
||||||
|
|
||||||
- **See** the engine state via virtual cameras, semantic segmentation maps, and profiler screenshots.
|
- **See** the engine state via rendered-frame screenshots, virtual cameras, semantic segmentation maps, and profiler screenshots.
|
||||||
- **Read** the complete ECS world state through native JSON serialization.
|
- **Read** the complete ECS world state through native JSON serialization.
|
||||||
- **Modify** the running engine via declarative JSON commands and, in Development Mode, via hot-reloaded C# scripts.
|
- **Modify** the running engine via declarative JSON commands and, in Development Mode, via hot-reloaded C# scripts.
|
||||||
|
|
||||||
The architecture prioritizes **production maturity** over experimental technologies: Vulkan (via Silk.NET.Vulkan), Flecs.NET (C# bindings for the C-based Flecs ECS), SDL3-cs (ppy.SDL3-CS), and ImGui.NET/Hexa.NET.ImGui with a native Vulkan backend.
|
The architecture prioritizes **production maturity** over experimental technologies: Vulkan (via Silk.NET.Vulkan), Flecs.NET (C# bindings for the C-based Flecs ECS), SDL3-cs (ppy.SDL3-CS), and Hexa.NET.ImGui with a native Vulkan backend.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -112,12 +112,14 @@ All Roslyn and `AssemblyLoadContext` code is wrapped in `#if DEV_MODE`.
|
|||||||
|
|
||||||
### 3.3 Graphics HAL
|
### 3.3 Graphics HAL
|
||||||
|
|
||||||
**Vulkan via `Vortice.Vulkan`**
|
**Vulkan via `Silk.NET.Vulkan`**
|
||||||
|
|
||||||
- NuGet: `Vortice.Vulkan` 3.2.3
|
- NuGet: `Silk.NET.Vulkan` 2.21.0
|
||||||
- .NET 9/10 low-level bindings
|
- .NET 9/10 low-level bindings
|
||||||
- Includes VulkanMemoryAllocator, SPIRV-Cross, shaderc
|
- Mature, used by Silk.NET ecosystem
|
||||||
- Mature, MIT licensed, listed on vulkan.org
|
- MoltenVK provides macOS/iOS support
|
||||||
|
|
||||||
|
**Note:** Initial prototype used Vortice.Vulkan, but its loader segfaulted on the Kubuntu development setup. Silk.NET.Vulkan is the verified working binding.
|
||||||
|
|
||||||
**Why Vulkan over WebGPU:**
|
**Why Vulkan over WebGPU:**
|
||||||
|
|
||||||
@@ -426,19 +428,30 @@ Non-critical captures can be spread across multiple frames to avoid stuttering.
|
|||||||
|
|
||||||
The `AiGateway` is the only allowed path for the AI to modify the running engine. It prevents memory corruption, invalid state, and unsafe code execution.
|
The `AiGateway` is the only allowed path for the AI to modify the running engine. It prevents memory corruption, invalid state, and unsafe code execution.
|
||||||
|
|
||||||
### 6.2 Command Pattern (Works in Both Dev and Release)
|
### 6.2 MCP Server (Dev / Release)
|
||||||
|
|
||||||
The AI issues declarative JSON commands:
|
In Debug and Release configurations, the engine hosts an in-process **Model Context Protocol (MCP)** HTTP server. AI clients (Claude Desktop, Cursor, VS Code Copilot) can connect to it and call tools:
|
||||||
|
|
||||||
|
- `spawn_model` — spawn a named entity from a model file.
|
||||||
|
- `set_transform` — update entity position, rotation, scale.
|
||||||
|
- `delete_entity` — delete an entity by name.
|
||||||
|
- `list_entities` — list all named entities with a `Transform`.
|
||||||
|
- `capture_screenshot` — save a PNG of the current frame.
|
||||||
|
|
||||||
|
Commands are queued and executed on the main engine thread so the Flecs world is never touched from a background thread.
|
||||||
|
|
||||||
|
### 6.3 Command Pattern (Works in Both Dev and Release)
|
||||||
|
|
||||||
|
The AI can also issue declarative JSON commands directly:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"command": "spawn",
|
"type": "spawn_model",
|
||||||
"type": "enemy",
|
"name": "Enemy",
|
||||||
"at": [10, 0, 5],
|
"modelPath": "Models/enemy.obj",
|
||||||
"components": {
|
"position": [10, 0, 5],
|
||||||
"Transform": { "position": [10, 0, 5], "rotation": [0, 0, 0, 1], "scale": [1, 1, 1] },
|
"rotation": [0, 0, 0, 1],
|
||||||
"SemanticClass": { "classId": 1 }
|
"scale": [1, 1, 1]
|
||||||
}
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -447,10 +460,10 @@ Validation steps:
|
|||||||
1. JSON schema validation
|
1. JSON schema validation
|
||||||
2. Type existence check via Flecs reflection
|
2. Type existence check via Flecs reflection
|
||||||
3. Coordinate sanity check (e.g., no NaN, no extreme values)
|
3. Coordinate sanity check (e.g., no NaN, no extreme values)
|
||||||
4. Safe-name check (no `..` in prefab paths)
|
4. Safe-name check (no `..` in model paths)
|
||||||
5. Queue operation via `world.Defer()` for execution at the next frame boundary
|
5. Queue operation for execution at the next frame boundary
|
||||||
|
|
||||||
### 6.3 Scripting Validation (Dev Mode Only)
|
### 6.4 Scripting Validation (Dev Mode Only)
|
||||||
|
|
||||||
Before Roslyn compilation:
|
Before Roslyn compilation:
|
||||||
|
|
||||||
@@ -460,9 +473,9 @@ Before Roslyn compilation:
|
|||||||
4. **Reference validation**: Ensure all referenced types exist in the engine API surface.
|
4. **Reference validation**: Ensure all referenced types exist in the engine API surface.
|
||||||
5. **Sandboxed compilation**: Compile into isolated `AssemblyLoadContext`.
|
5. **Sandboxed compilation**: Compile into isolated `AssemblyLoadContext`.
|
||||||
|
|
||||||
### 6.4 Release Mode Limitation
|
### 6.5 Release Mode Limitation
|
||||||
|
|
||||||
In Release (NativeAOT), the AI cannot compile new C# code. It can only send JSON commands. This is a deliberate security and stability choice.
|
In Release (NativeAOT), the MCP server and ASP.NET Core are excluded. The AI cannot compile new C# code. It can only send JSON commands via `AiCommandProcessor`. This is a deliberate security and stability choice.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -488,10 +501,12 @@ In Release (NativeAOT), the AI cannot compile new C# code. It can only send JSON
|
|||||||
│ ├── Engine.Graphics/
|
│ ├── Engine.Graphics/
|
||||||
│ │ ├── VulkanContext.cs # Device, instance, queues
|
│ │ ├── VulkanContext.cs # Device, instance, queues
|
||||||
│ │ ├── Swapchain.cs # Swapchain management
|
│ │ ├── Swapchain.cs # Swapchain management
|
||||||
│ │ ├── RenderPassManager.cs # Dynamic rendering helpers
|
│ │ ├── MeshRenderer.cs # ECS mesh rendering
|
||||||
│ │ ├── SemanticRenderer.cs # Flat-color segmentation pass
|
│ │ ├── ScreenshotCapture.cs # Vulkan readback → PNG
|
||||||
│ │ ├── RttCamera.cs # Virtual camera + off-screen framebuffer
|
│ │ ├── VulkanPipeline.cs # Graphics pipeline
|
||||||
│ │ └── TextureReadback.cs # Image → staging buffer → JPEG
|
│ │ ├── VertexBuffer.cs # Vertex buffer helpers
|
||||||
|
│ │ ├── IndexBuffer.cs # Index buffer helpers
|
||||||
|
│ │ └── Loaders/ # ObjLoader, GltfLoader
|
||||||
│ │
|
│ │
|
||||||
│ ├── Engine.Diagnostics/
|
│ ├── Engine.Diagnostics/
|
||||||
│ │ ├── DiagnosticsManager.cs # Orchestrator
|
│ │ ├── DiagnosticsManager.cs # Orchestrator
|
||||||
@@ -500,12 +515,14 @@ In Release (NativeAOT), the AI cannot compile new C# code. It can only send JSON
|
|||||||
│ │ ├── Payload.cs # DiagnosticPayload class
|
│ │ ├── Payload.cs # DiagnosticPayload class
|
||||||
│ │ └── LogBuffer.cs # Circular console log buffer
|
│ │ └── LogBuffer.cs # Circular console log buffer
|
||||||
│ │
|
│ │
|
||||||
│ ├── Engine.AiGateway/
|
│ ├── Engine.AI/
|
||||||
│ │ ├── AiGateway.cs # Command parser + dispatcher
|
│ │ ├── AiCommandProcessor.cs # Parses and executes JSON commands
|
||||||
│ │ ├── CommandValidator.cs # JSON command validation
|
│ │ ├── AiCommandQueue.cs # Thread-safe command queue
|
||||||
│ │ ├── RoslynCompilerService.cs # Dev-Mode C# compilation
|
│ │ ├── Mcp/
|
||||||
│ │ ├── ScriptingSandbox.cs # AssemblyLoadContext isolation
|
│ │ │ ├── EngineMcpTools.cs # MCP tool definitions
|
||||||
│ │ └── TypeMigration.cs # Component type migration helper
|
│ │ │ └── McpEngineServerHost.cs # In-process MCP HTTP server
|
||||||
|
│ │ ├── Commands/ # AI command DTOs
|
||||||
|
│ │ └── Serialization/ # JSON converters for Vector3/Quaternion
|
||||||
│ │
|
│ │
|
||||||
│ ├── Engine.Editor/
|
│ ├── Engine.Editor/
|
||||||
│ │ ├── ImGuiController.cs # Hexa.NET.ImGui initialization
|
│ │ ├── ImGuiController.cs # Hexa.NET.ImGui initialization
|
||||||
@@ -595,14 +612,19 @@ In Release (NativeAOT), the AI cannot compile new C# code. It can only send JSON
|
|||||||
| Component | Package | Version | .NET | AOT | WASM | Mobile | Status |
|
| Component | Package | Version | .NET | AOT | WASM | Mobile | Status |
|
||||||
|-----------|---------|---------|------|-----|------|--------|--------|
|
|-----------|---------|---------|------|-----|------|--------|--------|
|
||||||
| SDL3 | `ppy.SDL3-CS` | 2026.520.0 | 9/10 | ✅ | ✅ | ✅ | Production |
|
| SDL3 | `ppy.SDL3-CS` | 2026.520.0 | 9/10 | ✅ | ✅ | ✅ | Production |
|
||||||
| Vulkan | `Vortice.Vulkan` | 3.2.3 | 9/10 | ✅ | ❌ | MoltenVK | Production |
|
| Vulkan | `Silk.NET.Vulkan` | 2.21.0 | 9/10 | ✅ | ❌ | MoltenVK | Production |
|
||||||
| ImGui | `Hexa.NET.ImGui` | latest | 9/10 | ✅ | ✅ | ✅ | Production |
|
| ImGui | `Hexa.NET.ImGui` | latest | 9/10 | ✅ | ✅ | ✅ | Production |
|
||||||
| ECS | `Flecs.NET.Release` | 4.0.3 | 8/9 | ✅* | ✅ | ✅ | Production |
|
| ECS | `Flecs.NET.Release` | 4.0.4-build.546 | 8/9 | ✅* | ✅ | ✅ | Production |
|
||||||
|
| Model loading | `SharpGLTF.Core` | 1.0.6 | 9/10 | ✅ | ✅ | ✅ | Production |
|
||||||
|
| AI bridge | `ModelContextProtocol` | 1.4.0 | 8/9 | ❌† | ✅ | ✅ | Production |
|
||||||
|
| Screenshot | `SixLabors.ImageSharp` | 3.1.11 | 9/10 | ✅ | ✅ | ✅ | Production |
|
||||||
| Physics | `JoltPhysicsSharp` | 2.21.0 | 9/10 | ✅ | ❌ | ✅ | Production |
|
| Physics | `JoltPhysicsSharp` | 2.21.0 | 9/10 | ✅ | ❌ | ✅ | Production |
|
||||||
| JPEG | `SixLabors.ImageSharp` | latest | 9/10 | ✅ | ✅ | ✅ | Production |
|
| JPEG | `SixLabors.ImageSharp` | latest | 9/10 | ✅ | ✅ | ✅ | Production |
|
||||||
|
|
||||||
\* Via `<FlecsStaticLink>true</FlecsStaticLink>`
|
\* Via `<FlecsStaticLink>true</FlecsStaticLink>`
|
||||||
|
|
||||||
|
† MCP server is excluded from `ReleaseAOT` because it depends on ASP.NET Core. JSON-only AI commands still work in AOT via `AiCommandProcessor`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10. KNOWN RISKS & MITIGATIONS
|
## 10. KNOWN RISKS & MITIGATIONS
|
||||||
@@ -628,10 +650,11 @@ You are coding for Cortex Engine, a C# (.NET 9) AI-Native multiplatform 3D game
|
|||||||
Stack:
|
Stack:
|
||||||
- C# .NET 9 with dual-runtime: JIT (Debug) for Roslyn hot-reload, NativeAOT (ReleaseAOT) for JSON-only AI commands
|
- C# .NET 9 with dual-runtime: JIT (Debug) for Roslyn hot-reload, NativeAOT (ReleaseAOT) for JSON-only AI commands
|
||||||
- SDL3-cs (ppy.SDL3-CS) for windowing and input
|
- SDL3-cs (ppy.SDL3-CS) for windowing and input
|
||||||
- Vortice.Vulkan for graphics
|
- Silk.NET.Vulkan for graphics
|
||||||
- Flecs.NET for ECS
|
- Flecs.NET for ECS
|
||||||
- Hexa.NET.ImGui for editor UI
|
- Hexa.NET.ImGui for editor UI
|
||||||
- JoltPhysicsSharp for physics
|
- JoltPhysicsSharp for physics
|
||||||
|
- ModelContextProtocol for AI tool integration
|
||||||
|
|
||||||
Rules:
|
Rules:
|
||||||
1. All state lives in ECS components. GameObject is a struct facade.
|
1. All state lives in ECS components. GameObject is a struct facade.
|
||||||
|
|||||||
@@ -29,12 +29,12 @@ class Program
|
|||||||
using var swapchain = new Swapchain(vulkan);
|
using var swapchain = new Swapchain(vulkan);
|
||||||
using var renderer = new MeshRenderer(vulkan, swapchain);
|
using var renderer = new MeshRenderer(vulkan, swapchain);
|
||||||
|
|
||||||
var processor = new AiCommandProcessor(world, LoadModel);
|
|
||||||
var queue = new AiCommandQueue(processor);
|
|
||||||
|
|
||||||
var (modelPath, mcpPort) = ParseArgs(args);
|
var (modelPath, mcpPort) = ParseArgs(args);
|
||||||
var mesh = LoadModel(modelPath);
|
var mesh = LoadModel(modelPath);
|
||||||
|
|
||||||
|
var processor = new AiCommandProcessor(world, LoadModel, path => renderer.RequestScreenshot(path));
|
||||||
|
var queue = new AiCommandQueue(processor);
|
||||||
|
|
||||||
var model = world.Entity("Model")
|
var model = world.Entity("Model")
|
||||||
.Set(new Transform(new Vector3(0.0f, 0.0f, 0.0f), Quaternion.Identity, new Vector3(0.5f)))
|
.Set(new Transform(new Vector3(0.0f, 0.0f, 0.0f), Quaternion.Identity, new Vector3(0.5f)))
|
||||||
.Set(mesh);
|
.Set(mesh);
|
||||||
@@ -55,6 +55,7 @@ class Program
|
|||||||
Console.WriteLine(processor.Process("""{ "type": "spawn_model", "name": "SecondCube", "modelPath": "Content/cube.obj", "position": [0.8, 0, 0], "scale": [0.3, 0.3, 0.3] }""").Message);
|
Console.WriteLine(processor.Process("""{ "type": "spawn_model", "name": "SecondCube", "modelPath": "Content/cube.obj", "position": [0.8, 0, 0], "scale": [0.3, 0.3, 0.3] }""").Message);
|
||||||
Console.WriteLine(processor.Process("""{ "type": "list_entities" }""").Message);
|
Console.WriteLine(processor.Process("""{ "type": "list_entities" }""").Message);
|
||||||
Console.WriteLine(processor.Process("""{ "type": "set_transform", "name": "SecondCube", "position": [0.8, 0.5, 0], "rotation": [0, 0, 0, 1], "scale": [0.3, 0.3, 0.3] }""").Message);
|
Console.WriteLine(processor.Process("""{ "type": "set_transform", "name": "SecondCube", "position": [0.8, 0.5, 0], "rotation": [0, 0, 0, 1], "scale": [0.3, 0.3, 0.3] }""").Message);
|
||||||
|
Console.WriteLine(processor.Process("""{ "type": "capture_screenshot", "outputPath": "Screenshots/demo.png" }""").Message);
|
||||||
|
|
||||||
#if !RELEASE_AOT
|
#if !RELEASE_AOT
|
||||||
// Start the MCP server in the background so AI agents can connect via HTTP.
|
// Start the MCP server in the background so AI agents can connect via HTTP.
|
||||||
|
|||||||
@@ -16,13 +16,15 @@ public sealed class AiCommandProcessor
|
|||||||
{
|
{
|
||||||
private readonly World _world;
|
private readonly World _world;
|
||||||
private readonly Func<string, Mesh> _modelLoader;
|
private readonly Func<string, Mesh> _modelLoader;
|
||||||
|
private readonly Action<string> _requestScreenshot;
|
||||||
|
|
||||||
public JsonSerializerOptions JsonOptions { get; }
|
public JsonSerializerOptions JsonOptions { get; }
|
||||||
|
|
||||||
public AiCommandProcessor(World world, Func<string, Mesh> modelLoader)
|
public AiCommandProcessor(World world, Func<string, Mesh> modelLoader, Action<string> requestScreenshot)
|
||||||
{
|
{
|
||||||
_world = world;
|
_world = world;
|
||||||
_modelLoader = modelLoader;
|
_modelLoader = modelLoader;
|
||||||
|
_requestScreenshot = requestScreenshot;
|
||||||
JsonOptions = new JsonSerializerOptions
|
JsonOptions = new JsonSerializerOptions
|
||||||
{
|
{
|
||||||
PropertyNameCaseInsensitive = true,
|
PropertyNameCaseInsensitive = true,
|
||||||
@@ -52,6 +54,7 @@ public sealed class AiCommandProcessor
|
|||||||
SetTransformCommand c => SetTransform(c),
|
SetTransformCommand c => SetTransform(c),
|
||||||
DeleteEntityCommand c => DeleteEntity(c),
|
DeleteEntityCommand c => DeleteEntity(c),
|
||||||
ListEntitiesCommand => ListEntities(),
|
ListEntitiesCommand => ListEntities(),
|
||||||
|
CaptureScreenshotCommand c => CaptureScreenshot(c),
|
||||||
_ => AiCommandResult.Error($"Unknown command type: {command.Type}")
|
_ => AiCommandResult.Error($"Unknown command type: {command.Type}")
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -122,4 +125,11 @@ public sealed class AiCommandProcessor
|
|||||||
var json = JsonSerializer.Serialize(names, JsonOptions);
|
var json = JsonSerializer.Serialize(names, JsonOptions);
|
||||||
return AiCommandResult.Ok(json);
|
return AiCommandResult.Ok(json);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private AiCommandResult CaptureScreenshot(CaptureScreenshotCommand command)
|
||||||
|
{
|
||||||
|
var path = command.OutputPath ?? $"screenshot_{DateTime.UtcNow:yyyyMMdd_HHmmss_fff}.png";
|
||||||
|
_requestScreenshot(path);
|
||||||
|
return AiCommandResult.Ok($"Screenshot requested: {path}");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ namespace Engine.AI.Commands;
|
|||||||
[JsonDerivedType(typeof(SetTransformCommand), "set_transform")]
|
[JsonDerivedType(typeof(SetTransformCommand), "set_transform")]
|
||||||
[JsonDerivedType(typeof(DeleteEntityCommand), "delete_entity")]
|
[JsonDerivedType(typeof(DeleteEntityCommand), "delete_entity")]
|
||||||
[JsonDerivedType(typeof(ListEntitiesCommand), "list_entities")]
|
[JsonDerivedType(typeof(ListEntitiesCommand), "list_entities")]
|
||||||
|
[JsonDerivedType(typeof(CaptureScreenshotCommand), "capture_screenshot")]
|
||||||
public abstract record AiCommand
|
public abstract record AiCommand
|
||||||
{
|
{
|
||||||
public string Type => GetType().Name.Replace("Command", "").ToLowerInvariant();
|
public string Type => GetType().Name.Replace("Command", "").ToLowerInvariant();
|
||||||
|
|||||||
@@ -0,0 +1,12 @@
|
|||||||
|
namespace Engine.AI.Commands;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Request a screenshot of the current rendered frame for AI visual analysis.
|
||||||
|
/// </summary>
|
||||||
|
public sealed record CaptureScreenshotCommand : AiCommand
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Output file path. If omitted, the engine chooses a default path.
|
||||||
|
/// </summary>
|
||||||
|
public string? OutputPath { get; init; }
|
||||||
|
}
|
||||||
@@ -70,6 +70,13 @@ public sealed class EngineMcpTools
|
|||||||
return EnqueueAndReturnMessage(cmd);
|
return EnqueueAndReturnMessage(cmd);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[McpServerTool, Description("Capture a screenshot of the current rendered frame and save it to disk.")]
|
||||||
|
public Task<string> CaptureScreenshot([Description("Optional output file path (default: screenshot_<timestamp>.png)")] string? outputPath = null)
|
||||||
|
{
|
||||||
|
var cmd = new CaptureScreenshotCommand { OutputPath = outputPath };
|
||||||
|
return EnqueueAndReturnMessage(cmd);
|
||||||
|
}
|
||||||
|
|
||||||
private async Task<string> EnqueueAndReturnMessage(AiCommand command)
|
private async Task<string> EnqueueAndReturnMessage(AiCommand command)
|
||||||
{
|
{
|
||||||
var result = await _queue.EnqueueAsync(command).ConfigureAwait(false);
|
var result = await _queue.EnqueueAsync(command).ConfigureAwait(false);
|
||||||
|
|||||||
@@ -23,6 +23,7 @@
|
|||||||
<PackageReference Include="Flecs.NET.Debug" Version="4.0.4-build.546" Condition="'$(Configuration)' == 'Debug'" />
|
<PackageReference Include="Flecs.NET.Debug" Version="4.0.4-build.546" Condition="'$(Configuration)' == 'Debug'" />
|
||||||
<PackageReference Include="Flecs.NET.Release" Version="4.0.4-build.546" Condition="'$(Configuration)' == 'Release' OR '$(Configuration)' == 'ReleaseAOT'" />
|
<PackageReference Include="Flecs.NET.Release" Version="4.0.4-build.546" Condition="'$(Configuration)' == 'Release' OR '$(Configuration)' == 'ReleaseAOT'" />
|
||||||
<PackageReference Include="SharpGLTF.Core" Version="1.0.6" />
|
<PackageReference Include="SharpGLTF.Core" Version="1.0.6" />
|
||||||
|
<PackageReference Include="SixLabors.ImageSharp" Version="3.1.11" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ public sealed unsafe class MeshRenderer : IDisposable
|
|||||||
private readonly VulkanContext _context;
|
private readonly VulkanContext _context;
|
||||||
private readonly Swapchain _swapchain;
|
private readonly Swapchain _swapchain;
|
||||||
private readonly VulkanPipeline _pipeline;
|
private readonly VulkanPipeline _pipeline;
|
||||||
|
private readonly ScreenshotCapture _screenshot;
|
||||||
private readonly Dictionary<Entity, MeshBuffers> _buffers = new();
|
private readonly Dictionary<Entity, MeshBuffers> _buffers = new();
|
||||||
private CommandPool _commandPool;
|
private CommandPool _commandPool;
|
||||||
private CommandBuffer[] _commandBuffers = null!;
|
private CommandBuffer[] _commandBuffers = null!;
|
||||||
@@ -48,6 +49,7 @@ public sealed unsafe class MeshRenderer : IDisposable
|
|||||||
{
|
{
|
||||||
_context = context;
|
_context = context;
|
||||||
_swapchain = swapchain;
|
_swapchain = swapchain;
|
||||||
|
_screenshot = new ScreenshotCapture(context, swapchain);
|
||||||
|
|
||||||
_pipeline = new VulkanPipeline(context, swapchain);
|
_pipeline = new VulkanPipeline(context, swapchain);
|
||||||
CreateCommandPool();
|
CreateCommandPool();
|
||||||
@@ -118,6 +120,10 @@ public sealed unsafe class MeshRenderer : IDisposable
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public void RequestScreenshot(string outputPath) => _screenshot.Request(outputPath);
|
||||||
|
|
||||||
|
public bool IsScreenshotRequested => _screenshot.IsRequested;
|
||||||
|
|
||||||
public void RenderWorld(World world)
|
public void RenderWorld(World world)
|
||||||
{
|
{
|
||||||
var frame = _currentFrame % 2;
|
var frame = _currentFrame % 2;
|
||||||
@@ -198,6 +204,10 @@ public sealed unsafe class MeshRenderer : IDisposable
|
|||||||
});
|
});
|
||||||
|
|
||||||
_context.Vk.CmdEndRenderPass(cmd);
|
_context.Vk.CmdEndRenderPass(cmd);
|
||||||
|
|
||||||
|
var swapchainImage = _swapchain.GetImage(imageIndex);
|
||||||
|
_screenshot.RecordReadback(cmd, swapchainImage, _swapchain.Extent.Width, _swapchain.Extent.Height, _swapchain.SurfaceFormat);
|
||||||
|
|
||||||
_context.Vk.EndCommandBuffer(cmd);
|
_context.Vk.EndCommandBuffer(cmd);
|
||||||
|
|
||||||
var waitSemaphore = _imageAvailableSemaphores[frame];
|
var waitSemaphore = _imageAvailableSemaphores[frame];
|
||||||
@@ -229,6 +239,14 @@ public sealed unsafe class MeshRenderer : IDisposable
|
|||||||
};
|
};
|
||||||
|
|
||||||
_context.KhrSwapchain!.QueuePresent(_context.PresentQueue, &presentInfo);
|
_context.KhrSwapchain!.QueuePresent(_context.PresentQueue, &presentInfo);
|
||||||
|
|
||||||
|
// If a screenshot was requested, wait for the GPU to finish the readback and save the file.
|
||||||
|
if (_screenshot.IsRequested)
|
||||||
|
{
|
||||||
|
_context.Vk.WaitForFences(_context.Device, 1, &fence, true, ulong.MaxValue);
|
||||||
|
_screenshot.Save(_swapchain.Extent.Width, _swapchain.Extent.Height, _swapchain.SurfaceFormat);
|
||||||
|
}
|
||||||
|
|
||||||
_currentFrame++;
|
_currentFrame++;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -308,6 +326,8 @@ public sealed unsafe class MeshRenderer : IDisposable
|
|||||||
{
|
{
|
||||||
_context.Vk.DeviceWaitIdle(_context.Device);
|
_context.Vk.DeviceWaitIdle(_context.Device);
|
||||||
|
|
||||||
|
_screenshot.Dispose();
|
||||||
|
|
||||||
foreach (var buffers in _buffers.Values)
|
foreach (var buffers in _buffers.Values)
|
||||||
buffers.Dispose();
|
buffers.Dispose();
|
||||||
_buffers.Clear();
|
_buffers.Clear();
|
||||||
|
|||||||
@@ -0,0 +1,287 @@
|
|||||||
|
using System;
|
||||||
|
using System.IO;
|
||||||
|
using Silk.NET.Core;
|
||||||
|
using Silk.NET.Vulkan;
|
||||||
|
using SixLabors.ImageSharp;
|
||||||
|
using SixLabors.ImageSharp.PixelFormats;
|
||||||
|
|
||||||
|
namespace Engine.Graphics;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Captures the current swapchain image to a PNG file on disk.
|
||||||
|
/// Used by AI agents to visually inspect the running engine.
|
||||||
|
/// </summary>
|
||||||
|
public sealed unsafe class ScreenshotCapture : IDisposable
|
||||||
|
{
|
||||||
|
private readonly VulkanContext _context;
|
||||||
|
private readonly Swapchain _swapchain;
|
||||||
|
private Silk.NET.Vulkan.Buffer _stagingBuffer;
|
||||||
|
private DeviceMemory _stagingMemory;
|
||||||
|
private ulong _stagingSize;
|
||||||
|
private bool _requested;
|
||||||
|
private string _outputPath = string.Empty;
|
||||||
|
private bool _ready;
|
||||||
|
|
||||||
|
public ScreenshotCapture(VulkanContext context, Swapchain swapchain)
|
||||||
|
{
|
||||||
|
_context = context;
|
||||||
|
_swapchain = swapchain;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Request a screenshot to be captured on the next frame.
|
||||||
|
/// </summary>
|
||||||
|
public void Request(string outputPath)
|
||||||
|
{
|
||||||
|
_outputPath = outputPath;
|
||||||
|
_requested = true;
|
||||||
|
_ready = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// True if a screenshot has been requested but not yet saved.
|
||||||
|
/// </summary>
|
||||||
|
public bool IsRequested => _requested;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Records the image readback commands into the given command buffer.
|
||||||
|
/// Must be called after the render pass has ended and before the image is presented.
|
||||||
|
/// </summary>
|
||||||
|
public void RecordReadback(CommandBuffer cmd, Silk.NET.Vulkan.Image sourceImage, uint width, uint height, Format format)
|
||||||
|
{
|
||||||
|
if (!_requested)
|
||||||
|
return;
|
||||||
|
|
||||||
|
var pixelSize = GetPixelSize(format);
|
||||||
|
var rowPitch = width * pixelSize;
|
||||||
|
var imageSize = rowPitch * height;
|
||||||
|
|
||||||
|
EnsureStagingBuffer(imageSize);
|
||||||
|
|
||||||
|
// Transition from present layout to transfer source.
|
||||||
|
var barrier = new ImageMemoryBarrier
|
||||||
|
{
|
||||||
|
SType = StructureType.ImageMemoryBarrier,
|
||||||
|
OldLayout = ImageLayout.PresentSrcKhr,
|
||||||
|
NewLayout = ImageLayout.TransferSrcOptimal,
|
||||||
|
SrcAccessMask = AccessFlags.None,
|
||||||
|
DstAccessMask = AccessFlags.TransferReadBit,
|
||||||
|
Image = sourceImage,
|
||||||
|
SubresourceRange = new ImageSubresourceRange(ImageAspectFlags.ColorBit, 0, 1, 0, 1)
|
||||||
|
};
|
||||||
|
|
||||||
|
_context.Vk.CmdPipelineBarrier(cmd, PipelineStageFlags.TransferBit, PipelineStageFlags.TransferBit, 0, 0, null, 0, null, 1, &barrier);
|
||||||
|
|
||||||
|
var copyRegion = new BufferImageCopy
|
||||||
|
{
|
||||||
|
BufferOffset = 0,
|
||||||
|
BufferRowLength = 0,
|
||||||
|
BufferImageHeight = 0,
|
||||||
|
ImageSubresource = new ImageSubresourceLayers(ImageAspectFlags.ColorBit, 0, 0, 1),
|
||||||
|
ImageOffset = new Offset3D(0, 0, 0),
|
||||||
|
ImageExtent = new Extent3D(width, height, 1)
|
||||||
|
};
|
||||||
|
|
||||||
|
_context.Vk.CmdCopyImageToBuffer(cmd, sourceImage, ImageLayout.TransferSrcOptimal, _stagingBuffer, 1, ©Region);
|
||||||
|
|
||||||
|
// Transition back to present layout.
|
||||||
|
barrier.OldLayout = ImageLayout.TransferSrcOptimal;
|
||||||
|
barrier.NewLayout = ImageLayout.PresentSrcKhr;
|
||||||
|
barrier.SrcAccessMask = AccessFlags.TransferReadBit;
|
||||||
|
barrier.DstAccessMask = AccessFlags.None;
|
||||||
|
|
||||||
|
_context.Vk.CmdPipelineBarrier(cmd, PipelineStageFlags.TransferBit, PipelineStageFlags.TransferBit, 0, 0, null, 0, null, 1, &barrier);
|
||||||
|
|
||||||
|
_ready = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Save the captured pixels to disk. Must be called after the command buffer containing the readback has finished.
|
||||||
|
/// </summary>
|
||||||
|
public void Save(uint width, uint height, Format format)
|
||||||
|
{
|
||||||
|
if (!_ready)
|
||||||
|
return;
|
||||||
|
|
||||||
|
var pixelSize = GetPixelSize(format);
|
||||||
|
var rowPitch = width * pixelSize;
|
||||||
|
var imageSize = rowPitch * height;
|
||||||
|
|
||||||
|
void* mappedData;
|
||||||
|
var result = _context.Vk.MapMemory(_context.Device, _stagingMemory, 0, imageSize, MemoryMapFlags.None, &mappedData);
|
||||||
|
if (result != Result.Success)
|
||||||
|
throw new InvalidOperationException($"vkMapMemory failed: {result}");
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var directory = Path.GetDirectoryName(_outputPath);
|
||||||
|
if (!string.IsNullOrEmpty(directory))
|
||||||
|
Directory.CreateDirectory(directory);
|
||||||
|
|
||||||
|
SavePixels(mappedData, width, height, rowPitch, format);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
_context.Vk.UnmapMemory(_context.Device, _stagingMemory);
|
||||||
|
}
|
||||||
|
|
||||||
|
Console.WriteLine($"Screenshot saved: {_outputPath}");
|
||||||
|
_requested = false;
|
||||||
|
_ready = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void SavePixels(void* mappedData, uint width, uint height, uint rowPitch, Format format)
|
||||||
|
{
|
||||||
|
if (format == Format.B8G8R8A8Unorm || format == Format.B8G8R8A8Srgb)
|
||||||
|
{
|
||||||
|
SaveBgra(mappedData, width, height, rowPitch);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (format == Format.R8G8B8A8Unorm || format == Format.R8G8B8A8Srgb)
|
||||||
|
{
|
||||||
|
SaveRgba(mappedData, width, height, rowPitch);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new NotSupportedException($"Screenshot format {format} is not supported.");
|
||||||
|
}
|
||||||
|
|
||||||
|
private void SaveBgra(void* mappedData, uint width, uint height, uint rowPitch)
|
||||||
|
{
|
||||||
|
using var image = new SixLabors.ImageSharp.Image<Rgba32>((int)width, (int)height);
|
||||||
|
var src = (byte*)mappedData;
|
||||||
|
|
||||||
|
for (var y = 0; y < height; y++)
|
||||||
|
{
|
||||||
|
var rowStart = src + y * rowPitch;
|
||||||
|
for (var x = 0; x < width; x++)
|
||||||
|
{
|
||||||
|
var b = rowStart[x * 4 + 0];
|
||||||
|
var g = rowStart[x * 4 + 1];
|
||||||
|
var r = rowStart[x * 4 + 2];
|
||||||
|
var a = rowStart[x * 4 + 3];
|
||||||
|
image[x, y] = new Rgba32(r, g, b, a);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
image.SaveAsPng(_outputPath);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void SaveRgba(void* mappedData, uint width, uint height, uint rowPitch)
|
||||||
|
{
|
||||||
|
using var image = new SixLabors.ImageSharp.Image<Rgba32>((int)width, (int)height);
|
||||||
|
var src = (byte*)mappedData;
|
||||||
|
|
||||||
|
for (var y = 0; y < height; y++)
|
||||||
|
{
|
||||||
|
var rowStart = src + y * rowPitch;
|
||||||
|
for (var x = 0; x < width; x++)
|
||||||
|
{
|
||||||
|
var r = rowStart[x * 4 + 0];
|
||||||
|
var g = rowStart[x * 4 + 1];
|
||||||
|
var b = rowStart[x * 4 + 2];
|
||||||
|
var a = rowStart[x * 4 + 3];
|
||||||
|
image[x, y] = new Rgba32(r, g, b, a);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
image.SaveAsPng(_outputPath);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void EnsureStagingBuffer(ulong size)
|
||||||
|
{
|
||||||
|
if (_stagingSize >= size)
|
||||||
|
return;
|
||||||
|
|
||||||
|
if (_stagingBuffer.Handle != 0)
|
||||||
|
{
|
||||||
|
_context.Vk.DestroyBuffer(_context.Device, _stagingBuffer, null);
|
||||||
|
_context.Vk.FreeMemory(_context.Device, _stagingMemory, null);
|
||||||
|
}
|
||||||
|
|
||||||
|
_stagingSize = size;
|
||||||
|
_stagingBuffer = CreateBuffer(size, BufferUsageFlags.TransferDstBit);
|
||||||
|
var requirements = GetMemoryRequirements(_stagingBuffer);
|
||||||
|
_stagingMemory = AllocateMemory(requirements, MemoryPropertyFlags.HostVisibleBit | MemoryPropertyFlags.HostCoherentBit);
|
||||||
|
|
||||||
|
var result = _context.Vk.BindBufferMemory(_context.Device, _stagingBuffer, _stagingMemory, 0);
|
||||||
|
if (result != Result.Success)
|
||||||
|
throw new InvalidOperationException($"vkBindBufferMemory failed: {result}");
|
||||||
|
}
|
||||||
|
|
||||||
|
private Silk.NET.Vulkan.Buffer CreateBuffer(ulong size, BufferUsageFlags usage)
|
||||||
|
{
|
||||||
|
var createInfo = new BufferCreateInfo
|
||||||
|
{
|
||||||
|
SType = StructureType.BufferCreateInfo,
|
||||||
|
Size = size,
|
||||||
|
Usage = usage,
|
||||||
|
SharingMode = SharingMode.Exclusive
|
||||||
|
};
|
||||||
|
|
||||||
|
Silk.NET.Vulkan.Buffer buffer;
|
||||||
|
var result = _context.Vk.CreateBuffer(_context.Device, &createInfo, null, &buffer);
|
||||||
|
if (result != Result.Success)
|
||||||
|
throw new InvalidOperationException($"vkCreateBuffer failed: {result}");
|
||||||
|
return buffer;
|
||||||
|
}
|
||||||
|
|
||||||
|
private MemoryRequirements GetMemoryRequirements(Silk.NET.Vulkan.Buffer buffer)
|
||||||
|
{
|
||||||
|
MemoryRequirements requirements;
|
||||||
|
_context.Vk.GetBufferMemoryRequirements(_context.Device, buffer, &requirements);
|
||||||
|
return requirements;
|
||||||
|
}
|
||||||
|
|
||||||
|
private DeviceMemory AllocateMemory(MemoryRequirements requirements, MemoryPropertyFlags properties)
|
||||||
|
{
|
||||||
|
var memoryTypeIndex = FindMemoryType(requirements.MemoryTypeBits, properties);
|
||||||
|
var allocInfo = new MemoryAllocateInfo
|
||||||
|
{
|
||||||
|
SType = StructureType.MemoryAllocateInfo,
|
||||||
|
AllocationSize = requirements.Size,
|
||||||
|
MemoryTypeIndex = memoryTypeIndex
|
||||||
|
};
|
||||||
|
|
||||||
|
DeviceMemory memory;
|
||||||
|
var result = _context.Vk.AllocateMemory(_context.Device, &allocInfo, null, &memory);
|
||||||
|
if (result != Result.Success)
|
||||||
|
throw new InvalidOperationException($"vkAllocateMemory failed: {result}");
|
||||||
|
return memory;
|
||||||
|
}
|
||||||
|
|
||||||
|
private uint FindMemoryType(uint typeFilter, MemoryPropertyFlags properties)
|
||||||
|
{
|
||||||
|
PhysicalDeviceMemoryProperties memoryProperties;
|
||||||
|
_context.Vk.GetPhysicalDeviceMemoryProperties(_context.PhysicalDevice, &memoryProperties);
|
||||||
|
for (var i = 0; i < memoryProperties.MemoryTypeCount; i++)
|
||||||
|
{
|
||||||
|
if ((typeFilter & (1u << i)) != 0 &&
|
||||||
|
(memoryProperties.MemoryTypes[i].PropertyFlags & properties) == properties)
|
||||||
|
{
|
||||||
|
return (uint)i;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw new InvalidOperationException("Failed to find suitable memory type.");
|
||||||
|
}
|
||||||
|
|
||||||
|
private static uint GetPixelSize(Format format)
|
||||||
|
{
|
||||||
|
return format switch
|
||||||
|
{
|
||||||
|
Format.B8G8R8A8Unorm or Format.B8G8R8A8Srgb or Format.R8G8B8A8Unorm or Format.R8G8B8A8Srgb => 4,
|
||||||
|
_ => throw new NotSupportedException($"Format {format} is not supported for screenshots.")
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
if (_stagingBuffer.Handle != 0)
|
||||||
|
{
|
||||||
|
_context.Vk.DeviceWaitIdle(_context.Device);
|
||||||
|
_context.Vk.DestroyBuffer(_context.Device, _stagingBuffer, null);
|
||||||
|
_context.Vk.FreeMemory(_context.Device, _stagingMemory, null);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -30,6 +30,9 @@ public sealed unsafe class Swapchain : IDisposable
|
|||||||
public Extent2D Extent => _extent;
|
public Extent2D Extent => _extent;
|
||||||
public SwapchainKHR Handle => _swapchain;
|
public SwapchainKHR Handle => _swapchain;
|
||||||
public uint ImageCount => (uint)_images.Length;
|
public uint ImageCount => (uint)_images.Length;
|
||||||
|
public Format SurfaceFormat => _surfaceFormat.Format;
|
||||||
|
|
||||||
|
public Image GetImage(uint index) => _images[index];
|
||||||
|
|
||||||
public Swapchain(VulkanContext context)
|
public Swapchain(VulkanContext context)
|
||||||
{
|
{
|
||||||
|
|||||||
Reference in New Issue
Block a user