feat: AI screenshot capture for visual analysis

- Add ScreenshotCapture class to Engine.Graphics using Vulkan image readback.
- Integrate screenshot capture into MeshRenderer; request triggers readback on next frame.
- Add SixLabors.ImageSharp 3.1.11 for PNG encoding (patched for CVE-2025-54575).
- Add capture_screenshot AI command and MCP tool.
- Wire screenshot request into Program.cs with demo command.
- Expose swapchain surface format and image accessor for readback.
- Add Screenshots/ to .gitignore.
- Update CORTEX_ENGINE_ARCHITECTURE.md with MCP, Silk.NET.Vulkan, ImageSharp, and screenshot capture.
This commit is contained in:
emil28092005
2026-06-16 20:09:57 +03:00
parent a9c783d204
commit 9312810f0c
11 changed files with 405 additions and 37 deletions
+3
View File
@@ -15,6 +15,9 @@ obj/
.DS_Store .DS_Store
Thumbs.db Thumbs.db
# Generated screenshots
Screenshots/
# dotnet # dotnet
*.dll *.dll
*.exe *.exe
+56 -33
View File
@@ -11,11 +11,11 @@
Cortex Engine is a 3D game engine built from scratch to provide a Unity-like development experience (GameObject/Component paradigm, Inspector, Hierarchy, Scene View) while being deeply integrated with Multimodal Large Language Models (MMLMs). The engine allows an AI to: Cortex Engine is a 3D game engine built from scratch to provide a Unity-like development experience (GameObject/Component paradigm, Inspector, Hierarchy, Scene View) while being deeply integrated with Multimodal Large Language Models (MMLMs). The engine allows an AI to:
- **See** the engine state via virtual cameras, semantic segmentation maps, and profiler screenshots. - **See** the engine state via rendered-frame screenshots, virtual cameras, semantic segmentation maps, and profiler screenshots.
- **Read** the complete ECS world state through native JSON serialization. - **Read** the complete ECS world state through native JSON serialization.
- **Modify** the running engine via declarative JSON commands and, in Development Mode, via hot-reloaded C# scripts. - **Modify** the running engine via declarative JSON commands and, in Development Mode, via hot-reloaded C# scripts.
The architecture prioritizes **production maturity** over experimental technologies: Vulkan (via Silk.NET.Vulkan), Flecs.NET (C# bindings for the C-based Flecs ECS), SDL3-cs (ppy.SDL3-CS), and ImGui.NET/Hexa.NET.ImGui with a native Vulkan backend. The architecture prioritizes **production maturity** over experimental technologies: Vulkan (via Silk.NET.Vulkan), Flecs.NET (C# bindings for the C-based Flecs ECS), SDL3-cs (ppy.SDL3-CS), and Hexa.NET.ImGui with a native Vulkan backend.
--- ---
@@ -112,12 +112,14 @@ All Roslyn and `AssemblyLoadContext` code is wrapped in `#if DEV_MODE`.
### 3.3 Graphics HAL ### 3.3 Graphics HAL
**Vulkan via `Vortice.Vulkan`** **Vulkan via `Silk.NET.Vulkan`**
- NuGet: `Vortice.Vulkan` 3.2.3 - NuGet: `Silk.NET.Vulkan` 2.21.0
- .NET 9/10 low-level bindings - .NET 9/10 low-level bindings
- Includes VulkanMemoryAllocator, SPIRV-Cross, shaderc - Mature, used by Silk.NET ecosystem
- Mature, MIT licensed, listed on vulkan.org - MoltenVK provides macOS/iOS support
**Note:** Initial prototype used Vortice.Vulkan, but its loader segfaulted on the Kubuntu development setup. Silk.NET.Vulkan is the verified working binding.
**Why Vulkan over WebGPU:** **Why Vulkan over WebGPU:**
@@ -426,19 +428,30 @@ Non-critical captures can be spread across multiple frames to avoid stuttering.
The `AiGateway` is the only allowed path for the AI to modify the running engine. It prevents memory corruption, invalid state, and unsafe code execution. The `AiGateway` is the only allowed path for the AI to modify the running engine. It prevents memory corruption, invalid state, and unsafe code execution.
### 6.2 Command Pattern (Works in Both Dev and Release) ### 6.2 MCP Server (Dev / Release)
The AI issues declarative JSON commands: In Debug and Release configurations, the engine hosts an in-process **Model Context Protocol (MCP)** HTTP server. AI clients (Claude Desktop, Cursor, VS Code Copilot) can connect to it and call tools:
- `spawn_model` — spawn a named entity from a model file.
- `set_transform` — update entity position, rotation, scale.
- `delete_entity` — delete an entity by name.
- `list_entities` — list all named entities with a `Transform`.
- `capture_screenshot` — save a PNG of the current frame.
Commands are queued and executed on the main engine thread so the Flecs world is never touched from a background thread.
### 6.3 Command Pattern (Works in Both Dev and Release)
The AI can also issue declarative JSON commands directly:
```json ```json
{ {
"command": "spawn", "type": "spawn_model",
"type": "enemy", "name": "Enemy",
"at": [10, 0, 5], "modelPath": "Models/enemy.obj",
"components": { "position": [10, 0, 5],
"Transform": { "position": [10, 0, 5], "rotation": [0, 0, 0, 1], "scale": [1, 1, 1] }, "rotation": [0, 0, 0, 1],
"SemanticClass": { "classId": 1 } "scale": [1, 1, 1]
}
} }
``` ```
@@ -447,10 +460,10 @@ Validation steps:
1. JSON schema validation 1. JSON schema validation
2. Type existence check via Flecs reflection 2. Type existence check via Flecs reflection
3. Coordinate sanity check (e.g., no NaN, no extreme values) 3. Coordinate sanity check (e.g., no NaN, no extreme values)
4. Safe-name check (no `..` in prefab paths) 4. Safe-name check (no `..` in model paths)
5. Queue operation via `world.Defer()` for execution at the next frame boundary 5. Queue operation for execution at the next frame boundary
### 6.3 Scripting Validation (Dev Mode Only) ### 6.4 Scripting Validation (Dev Mode Only)
Before Roslyn compilation: Before Roslyn compilation:
@@ -460,9 +473,9 @@ Before Roslyn compilation:
4. **Reference validation**: Ensure all referenced types exist in the engine API surface. 4. **Reference validation**: Ensure all referenced types exist in the engine API surface.
5. **Sandboxed compilation**: Compile into isolated `AssemblyLoadContext`. 5. **Sandboxed compilation**: Compile into isolated `AssemblyLoadContext`.
### 6.4 Release Mode Limitation ### 6.5 Release Mode Limitation
In Release (NativeAOT), the AI cannot compile new C# code. It can only send JSON commands. This is a deliberate security and stability choice. In Release (NativeAOT), the MCP server and ASP.NET Core are excluded. The AI cannot compile new C# code. It can only send JSON commands via `AiCommandProcessor`. This is a deliberate security and stability choice.
--- ---
@@ -488,10 +501,12 @@ In Release (NativeAOT), the AI cannot compile new C# code. It can only send JSON
│ ├── Engine.Graphics/ │ ├── Engine.Graphics/
│ │ ├── VulkanContext.cs # Device, instance, queues │ │ ├── VulkanContext.cs # Device, instance, queues
│ │ ├── Swapchain.cs # Swapchain management │ │ ├── Swapchain.cs # Swapchain management
│ │ ├── RenderPassManager.cs # Dynamic rendering helpers │ │ ├── MeshRenderer.cs # ECS mesh rendering
│ │ ├── SemanticRenderer.cs # Flat-color segmentation pass │ │ ├── ScreenshotCapture.cs # Vulkan readback → PNG
│ │ ├── RttCamera.cs # Virtual camera + off-screen framebuffer │ │ ├── VulkanPipeline.cs # Graphics pipeline
│ │ ── TextureReadback.cs # Image → staging buffer → JPEG │ │ ── VertexBuffer.cs # Vertex buffer helpers
│ │ ├── IndexBuffer.cs # Index buffer helpers
│ │ └── Loaders/ # ObjLoader, GltfLoader
│ │ │ │
│ ├── Engine.Diagnostics/ │ ├── Engine.Diagnostics/
│ │ ├── DiagnosticsManager.cs # Orchestrator │ │ ├── DiagnosticsManager.cs # Orchestrator
@@ -500,12 +515,14 @@ In Release (NativeAOT), the AI cannot compile new C# code. It can only send JSON
│ │ ├── Payload.cs # DiagnosticPayload class │ │ ├── Payload.cs # DiagnosticPayload class
│ │ └── LogBuffer.cs # Circular console log buffer │ │ └── LogBuffer.cs # Circular console log buffer
│ │ │ │
│ ├── Engine.AiGateway/ │ ├── Engine.AI/
│ │ ├── AiGateway.cs # Command parser + dispatcher │ │ ├── AiCommandProcessor.cs # Parses and executes JSON commands
│ │ ├── CommandValidator.cs # JSON command validation │ │ ├── AiCommandQueue.cs # Thread-safe command queue
│ │ ├── RoslynCompilerService.cs # Dev-Mode C# compilation │ │ ├── Mcp/
│ │ ├── ScriptingSandbox.cs # AssemblyLoadContext isolation │ │ │ ├── EngineMcpTools.cs # MCP tool definitions
│ │ └── TypeMigration.cs # Component type migration helper │ │ │ └── McpEngineServerHost.cs # In-process MCP HTTP server
│ │ ├── Commands/ # AI command DTOs
│ │ └── Serialization/ # JSON converters for Vector3/Quaternion
│ │ │ │
│ ├── Engine.Editor/ │ ├── Engine.Editor/
│ │ ├── ImGuiController.cs # Hexa.NET.ImGui initialization │ │ ├── ImGuiController.cs # Hexa.NET.ImGui initialization
@@ -595,14 +612,19 @@ In Release (NativeAOT), the AI cannot compile new C# code. It can only send JSON
| Component | Package | Version | .NET | AOT | WASM | Mobile | Status | | Component | Package | Version | .NET | AOT | WASM | Mobile | Status |
|-----------|---------|---------|------|-----|------|--------|--------| |-----------|---------|---------|------|-----|------|--------|--------|
| SDL3 | `ppy.SDL3-CS` | 2026.520.0 | 9/10 | ✅ | ✅ | ✅ | Production | | SDL3 | `ppy.SDL3-CS` | 2026.520.0 | 9/10 | ✅ | ✅ | ✅ | Production |
| Vulkan | `Vortice.Vulkan` | 3.2.3 | 9/10 | ✅ | ❌ | MoltenVK | Production | | Vulkan | `Silk.NET.Vulkan` | 2.21.0 | 9/10 | ✅ | ❌ | MoltenVK | Production |
| ImGui | `Hexa.NET.ImGui` | latest | 9/10 | ✅ | ✅ | ✅ | Production | | ImGui | `Hexa.NET.ImGui` | latest | 9/10 | ✅ | ✅ | ✅ | Production |
| ECS | `Flecs.NET.Release` | 4.0.3 | 8/9 | ✅* | ✅ | ✅ | Production | | ECS | `Flecs.NET.Release` | 4.0.4-build.546 | 8/9 | ✅* | ✅ | ✅ | Production |
| Model loading | `SharpGLTF.Core` | 1.0.6 | 9/10 | ✅ | ✅ | ✅ | Production |
| AI bridge | `ModelContextProtocol` | 1.4.0 | 8/9 | ❌† | ✅ | ✅ | Production |
| Screenshot | `SixLabors.ImageSharp` | 3.1.11 | 9/10 | ✅ | ✅ | ✅ | Production |
| Physics | `JoltPhysicsSharp` | 2.21.0 | 9/10 | ✅ | ❌ | ✅ | Production | | Physics | `JoltPhysicsSharp` | 2.21.0 | 9/10 | ✅ | ❌ | ✅ | Production |
| JPEG | `SixLabors.ImageSharp` | latest | 9/10 | ✅ | ✅ | ✅ | Production | | JPEG | `SixLabors.ImageSharp` | latest | 9/10 | ✅ | ✅ | ✅ | Production |
\* Via `<FlecsStaticLink>true</FlecsStaticLink>` \* Via `<FlecsStaticLink>true</FlecsStaticLink>`
† MCP server is excluded from `ReleaseAOT` because it depends on ASP.NET Core. JSON-only AI commands still work in AOT via `AiCommandProcessor`.
--- ---
## 10. KNOWN RISKS & MITIGATIONS ## 10. KNOWN RISKS & MITIGATIONS
@@ -628,10 +650,11 @@ You are coding for Cortex Engine, a C# (.NET 9) AI-Native multiplatform 3D game
Stack: Stack:
- C# .NET 9 with dual-runtime: JIT (Debug) for Roslyn hot-reload, NativeAOT (ReleaseAOT) for JSON-only AI commands - C# .NET 9 with dual-runtime: JIT (Debug) for Roslyn hot-reload, NativeAOT (ReleaseAOT) for JSON-only AI commands
- SDL3-cs (ppy.SDL3-CS) for windowing and input - SDL3-cs (ppy.SDL3-CS) for windowing and input
- Vortice.Vulkan for graphics - Silk.NET.Vulkan for graphics
- Flecs.NET for ECS - Flecs.NET for ECS
- Hexa.NET.ImGui for editor UI - Hexa.NET.ImGui for editor UI
- JoltPhysicsSharp for physics - JoltPhysicsSharp for physics
- ModelContextProtocol for AI tool integration
Rules: Rules:
1. All state lives in ECS components. GameObject is a struct facade. 1. All state lives in ECS components. GameObject is a struct facade.
+4 -3
View File
@@ -29,12 +29,12 @@ class Program
using var swapchain = new Swapchain(vulkan); using var swapchain = new Swapchain(vulkan);
using var renderer = new MeshRenderer(vulkan, swapchain); using var renderer = new MeshRenderer(vulkan, swapchain);
var processor = new AiCommandProcessor(world, LoadModel);
var queue = new AiCommandQueue(processor);
var (modelPath, mcpPort) = ParseArgs(args); var (modelPath, mcpPort) = ParseArgs(args);
var mesh = LoadModel(modelPath); var mesh = LoadModel(modelPath);
var processor = new AiCommandProcessor(world, LoadModel, path => renderer.RequestScreenshot(path));
var queue = new AiCommandQueue(processor);
var model = world.Entity("Model") var model = world.Entity("Model")
.Set(new Transform(new Vector3(0.0f, 0.0f, 0.0f), Quaternion.Identity, new Vector3(0.5f))) .Set(new Transform(new Vector3(0.0f, 0.0f, 0.0f), Quaternion.Identity, new Vector3(0.5f)))
.Set(mesh); .Set(mesh);
@@ -55,6 +55,7 @@ class Program
Console.WriteLine(processor.Process("""{ "type": "spawn_model", "name": "SecondCube", "modelPath": "Content/cube.obj", "position": [0.8, 0, 0], "scale": [0.3, 0.3, 0.3] }""").Message); Console.WriteLine(processor.Process("""{ "type": "spawn_model", "name": "SecondCube", "modelPath": "Content/cube.obj", "position": [0.8, 0, 0], "scale": [0.3, 0.3, 0.3] }""").Message);
Console.WriteLine(processor.Process("""{ "type": "list_entities" }""").Message); Console.WriteLine(processor.Process("""{ "type": "list_entities" }""").Message);
Console.WriteLine(processor.Process("""{ "type": "set_transform", "name": "SecondCube", "position": [0.8, 0.5, 0], "rotation": [0, 0, 0, 1], "scale": [0.3, 0.3, 0.3] }""").Message); Console.WriteLine(processor.Process("""{ "type": "set_transform", "name": "SecondCube", "position": [0.8, 0.5, 0], "rotation": [0, 0, 0, 1], "scale": [0.3, 0.3, 0.3] }""").Message);
Console.WriteLine(processor.Process("""{ "type": "capture_screenshot", "outputPath": "Screenshots/demo.png" }""").Message);
#if !RELEASE_AOT #if !RELEASE_AOT
// Start the MCP server in the background so AI agents can connect via HTTP. // Start the MCP server in the background so AI agents can connect via HTTP.
+11 -1
View File
@@ -16,13 +16,15 @@ public sealed class AiCommandProcessor
{ {
private readonly World _world; private readonly World _world;
private readonly Func<string, Mesh> _modelLoader; private readonly Func<string, Mesh> _modelLoader;
private readonly Action<string> _requestScreenshot;
public JsonSerializerOptions JsonOptions { get; } public JsonSerializerOptions JsonOptions { get; }
public AiCommandProcessor(World world, Func<string, Mesh> modelLoader) public AiCommandProcessor(World world, Func<string, Mesh> modelLoader, Action<string> requestScreenshot)
{ {
_world = world; _world = world;
_modelLoader = modelLoader; _modelLoader = modelLoader;
_requestScreenshot = requestScreenshot;
JsonOptions = new JsonSerializerOptions JsonOptions = new JsonSerializerOptions
{ {
PropertyNameCaseInsensitive = true, PropertyNameCaseInsensitive = true,
@@ -52,6 +54,7 @@ public sealed class AiCommandProcessor
SetTransformCommand c => SetTransform(c), SetTransformCommand c => SetTransform(c),
DeleteEntityCommand c => DeleteEntity(c), DeleteEntityCommand c => DeleteEntity(c),
ListEntitiesCommand => ListEntities(), ListEntitiesCommand => ListEntities(),
CaptureScreenshotCommand c => CaptureScreenshot(c),
_ => AiCommandResult.Error($"Unknown command type: {command.Type}") _ => AiCommandResult.Error($"Unknown command type: {command.Type}")
}; };
} }
@@ -122,4 +125,11 @@ public sealed class AiCommandProcessor
var json = JsonSerializer.Serialize(names, JsonOptions); var json = JsonSerializer.Serialize(names, JsonOptions);
return AiCommandResult.Ok(json); return AiCommandResult.Ok(json);
} }
private AiCommandResult CaptureScreenshot(CaptureScreenshotCommand command)
{
var path = command.OutputPath ?? $"screenshot_{DateTime.UtcNow:yyyyMMdd_HHmmss_fff}.png";
_requestScreenshot(path);
return AiCommandResult.Ok($"Screenshot requested: {path}");
}
} }
+1
View File
@@ -11,6 +11,7 @@ namespace Engine.AI.Commands;
[JsonDerivedType(typeof(SetTransformCommand), "set_transform")] [JsonDerivedType(typeof(SetTransformCommand), "set_transform")]
[JsonDerivedType(typeof(DeleteEntityCommand), "delete_entity")] [JsonDerivedType(typeof(DeleteEntityCommand), "delete_entity")]
[JsonDerivedType(typeof(ListEntitiesCommand), "list_entities")] [JsonDerivedType(typeof(ListEntitiesCommand), "list_entities")]
[JsonDerivedType(typeof(CaptureScreenshotCommand), "capture_screenshot")]
public abstract record AiCommand public abstract record AiCommand
{ {
public string Type => GetType().Name.Replace("Command", "").ToLowerInvariant(); public string Type => GetType().Name.Replace("Command", "").ToLowerInvariant();
@@ -0,0 +1,12 @@
namespace Engine.AI.Commands;
/// <summary>
/// Request a screenshot of the current rendered frame for AI visual analysis.
/// </summary>
public sealed record CaptureScreenshotCommand : AiCommand
{
/// <summary>
/// Output file path. If omitted, the engine chooses a default path.
/// </summary>
public string? OutputPath { get; init; }
}
+7
View File
@@ -70,6 +70,13 @@ public sealed class EngineMcpTools
return EnqueueAndReturnMessage(cmd); return EnqueueAndReturnMessage(cmd);
} }
[McpServerTool, Description("Capture a screenshot of the current rendered frame and save it to disk.")]
public Task<string> CaptureScreenshot([Description("Optional output file path (default: screenshot_<timestamp>.png)")] string? outputPath = null)
{
var cmd = new CaptureScreenshotCommand { OutputPath = outputPath };
return EnqueueAndReturnMessage(cmd);
}
private async Task<string> EnqueueAndReturnMessage(AiCommand command) private async Task<string> EnqueueAndReturnMessage(AiCommand command)
{ {
var result = await _queue.EnqueueAsync(command).ConfigureAwait(false); var result = await _queue.EnqueueAsync(command).ConfigureAwait(false);
@@ -23,6 +23,7 @@
<PackageReference Include="Flecs.NET.Debug" Version="4.0.4-build.546" Condition="'$(Configuration)' == 'Debug'" /> <PackageReference Include="Flecs.NET.Debug" Version="4.0.4-build.546" Condition="'$(Configuration)' == 'Debug'" />
<PackageReference Include="Flecs.NET.Release" Version="4.0.4-build.546" Condition="'$(Configuration)' == 'Release' OR '$(Configuration)' == 'ReleaseAOT'" /> <PackageReference Include="Flecs.NET.Release" Version="4.0.4-build.546" Condition="'$(Configuration)' == 'Release' OR '$(Configuration)' == 'ReleaseAOT'" />
<PackageReference Include="SharpGLTF.Core" Version="1.0.6" /> <PackageReference Include="SharpGLTF.Core" Version="1.0.6" />
<PackageReference Include="SixLabors.ImageSharp" Version="3.1.11" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
+20
View File
@@ -18,6 +18,7 @@ public sealed unsafe class MeshRenderer : IDisposable
private readonly VulkanContext _context; private readonly VulkanContext _context;
private readonly Swapchain _swapchain; private readonly Swapchain _swapchain;
private readonly VulkanPipeline _pipeline; private readonly VulkanPipeline _pipeline;
private readonly ScreenshotCapture _screenshot;
private readonly Dictionary<Entity, MeshBuffers> _buffers = new(); private readonly Dictionary<Entity, MeshBuffers> _buffers = new();
private CommandPool _commandPool; private CommandPool _commandPool;
private CommandBuffer[] _commandBuffers = null!; private CommandBuffer[] _commandBuffers = null!;
@@ -48,6 +49,7 @@ public sealed unsafe class MeshRenderer : IDisposable
{ {
_context = context; _context = context;
_swapchain = swapchain; _swapchain = swapchain;
_screenshot = new ScreenshotCapture(context, swapchain);
_pipeline = new VulkanPipeline(context, swapchain); _pipeline = new VulkanPipeline(context, swapchain);
CreateCommandPool(); CreateCommandPool();
@@ -118,6 +120,10 @@ public sealed unsafe class MeshRenderer : IDisposable
} }
} }
public void RequestScreenshot(string outputPath) => _screenshot.Request(outputPath);
public bool IsScreenshotRequested => _screenshot.IsRequested;
public void RenderWorld(World world) public void RenderWorld(World world)
{ {
var frame = _currentFrame % 2; var frame = _currentFrame % 2;
@@ -198,6 +204,10 @@ public sealed unsafe class MeshRenderer : IDisposable
}); });
_context.Vk.CmdEndRenderPass(cmd); _context.Vk.CmdEndRenderPass(cmd);
var swapchainImage = _swapchain.GetImage(imageIndex);
_screenshot.RecordReadback(cmd, swapchainImage, _swapchain.Extent.Width, _swapchain.Extent.Height, _swapchain.SurfaceFormat);
_context.Vk.EndCommandBuffer(cmd); _context.Vk.EndCommandBuffer(cmd);
var waitSemaphore = _imageAvailableSemaphores[frame]; var waitSemaphore = _imageAvailableSemaphores[frame];
@@ -229,6 +239,14 @@ public sealed unsafe class MeshRenderer : IDisposable
}; };
_context.KhrSwapchain!.QueuePresent(_context.PresentQueue, &presentInfo); _context.KhrSwapchain!.QueuePresent(_context.PresentQueue, &presentInfo);
// If a screenshot was requested, wait for the GPU to finish the readback and save the file.
if (_screenshot.IsRequested)
{
_context.Vk.WaitForFences(_context.Device, 1, &fence, true, ulong.MaxValue);
_screenshot.Save(_swapchain.Extent.Width, _swapchain.Extent.Height, _swapchain.SurfaceFormat);
}
_currentFrame++; _currentFrame++;
} }
@@ -308,6 +326,8 @@ public sealed unsafe class MeshRenderer : IDisposable
{ {
_context.Vk.DeviceWaitIdle(_context.Device); _context.Vk.DeviceWaitIdle(_context.Device);
_screenshot.Dispose();
foreach (var buffers in _buffers.Values) foreach (var buffers in _buffers.Values)
buffers.Dispose(); buffers.Dispose();
_buffers.Clear(); _buffers.Clear();
+287
View File
@@ -0,0 +1,287 @@
using System;
using System.IO;
using Silk.NET.Core;
using Silk.NET.Vulkan;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
namespace Engine.Graphics;
/// <summary>
/// Captures the current swapchain image to a PNG file on disk.
/// Used by AI agents to visually inspect the running engine.
/// </summary>
public sealed unsafe class ScreenshotCapture : IDisposable
{
private readonly VulkanContext _context;
private readonly Swapchain _swapchain;
private Silk.NET.Vulkan.Buffer _stagingBuffer;
private DeviceMemory _stagingMemory;
private ulong _stagingSize;
private bool _requested;
private string _outputPath = string.Empty;
private bool _ready;
public ScreenshotCapture(VulkanContext context, Swapchain swapchain)
{
_context = context;
_swapchain = swapchain;
}
/// <summary>
/// Request a screenshot to be captured on the next frame.
/// </summary>
public void Request(string outputPath)
{
_outputPath = outputPath;
_requested = true;
_ready = false;
}
/// <summary>
/// True if a screenshot has been requested but not yet saved.
/// </summary>
public bool IsRequested => _requested;
/// <summary>
/// Records the image readback commands into the given command buffer.
/// Must be called after the render pass has ended and before the image is presented.
/// </summary>
public void RecordReadback(CommandBuffer cmd, Silk.NET.Vulkan.Image sourceImage, uint width, uint height, Format format)
{
if (!_requested)
return;
var pixelSize = GetPixelSize(format);
var rowPitch = width * pixelSize;
var imageSize = rowPitch * height;
EnsureStagingBuffer(imageSize);
// Transition from present layout to transfer source.
var barrier = new ImageMemoryBarrier
{
SType = StructureType.ImageMemoryBarrier,
OldLayout = ImageLayout.PresentSrcKhr,
NewLayout = ImageLayout.TransferSrcOptimal,
SrcAccessMask = AccessFlags.None,
DstAccessMask = AccessFlags.TransferReadBit,
Image = sourceImage,
SubresourceRange = new ImageSubresourceRange(ImageAspectFlags.ColorBit, 0, 1, 0, 1)
};
_context.Vk.CmdPipelineBarrier(cmd, PipelineStageFlags.TransferBit, PipelineStageFlags.TransferBit, 0, 0, null, 0, null, 1, &barrier);
var copyRegion = new BufferImageCopy
{
BufferOffset = 0,
BufferRowLength = 0,
BufferImageHeight = 0,
ImageSubresource = new ImageSubresourceLayers(ImageAspectFlags.ColorBit, 0, 0, 1),
ImageOffset = new Offset3D(0, 0, 0),
ImageExtent = new Extent3D(width, height, 1)
};
_context.Vk.CmdCopyImageToBuffer(cmd, sourceImage, ImageLayout.TransferSrcOptimal, _stagingBuffer, 1, &copyRegion);
// Transition back to present layout.
barrier.OldLayout = ImageLayout.TransferSrcOptimal;
barrier.NewLayout = ImageLayout.PresentSrcKhr;
barrier.SrcAccessMask = AccessFlags.TransferReadBit;
barrier.DstAccessMask = AccessFlags.None;
_context.Vk.CmdPipelineBarrier(cmd, PipelineStageFlags.TransferBit, PipelineStageFlags.TransferBit, 0, 0, null, 0, null, 1, &barrier);
_ready = true;
}
/// <summary>
/// Save the captured pixels to disk. Must be called after the command buffer containing the readback has finished.
/// </summary>
public void Save(uint width, uint height, Format format)
{
if (!_ready)
return;
var pixelSize = GetPixelSize(format);
var rowPitch = width * pixelSize;
var imageSize = rowPitch * height;
void* mappedData;
var result = _context.Vk.MapMemory(_context.Device, _stagingMemory, 0, imageSize, MemoryMapFlags.None, &mappedData);
if (result != Result.Success)
throw new InvalidOperationException($"vkMapMemory failed: {result}");
try
{
var directory = Path.GetDirectoryName(_outputPath);
if (!string.IsNullOrEmpty(directory))
Directory.CreateDirectory(directory);
SavePixels(mappedData, width, height, rowPitch, format);
}
finally
{
_context.Vk.UnmapMemory(_context.Device, _stagingMemory);
}
Console.WriteLine($"Screenshot saved: {_outputPath}");
_requested = false;
_ready = false;
}
private void SavePixels(void* mappedData, uint width, uint height, uint rowPitch, Format format)
{
if (format == Format.B8G8R8A8Unorm || format == Format.B8G8R8A8Srgb)
{
SaveBgra(mappedData, width, height, rowPitch);
return;
}
if (format == Format.R8G8B8A8Unorm || format == Format.R8G8B8A8Srgb)
{
SaveRgba(mappedData, width, height, rowPitch);
return;
}
throw new NotSupportedException($"Screenshot format {format} is not supported.");
}
private void SaveBgra(void* mappedData, uint width, uint height, uint rowPitch)
{
using var image = new SixLabors.ImageSharp.Image<Rgba32>((int)width, (int)height);
var src = (byte*)mappedData;
for (var y = 0; y < height; y++)
{
var rowStart = src + y * rowPitch;
for (var x = 0; x < width; x++)
{
var b = rowStart[x * 4 + 0];
var g = rowStart[x * 4 + 1];
var r = rowStart[x * 4 + 2];
var a = rowStart[x * 4 + 3];
image[x, y] = new Rgba32(r, g, b, a);
}
}
image.SaveAsPng(_outputPath);
}
private void SaveRgba(void* mappedData, uint width, uint height, uint rowPitch)
{
using var image = new SixLabors.ImageSharp.Image<Rgba32>((int)width, (int)height);
var src = (byte*)mappedData;
for (var y = 0; y < height; y++)
{
var rowStart = src + y * rowPitch;
for (var x = 0; x < width; x++)
{
var r = rowStart[x * 4 + 0];
var g = rowStart[x * 4 + 1];
var b = rowStart[x * 4 + 2];
var a = rowStart[x * 4 + 3];
image[x, y] = new Rgba32(r, g, b, a);
}
}
image.SaveAsPng(_outputPath);
}
private void EnsureStagingBuffer(ulong size)
{
if (_stagingSize >= size)
return;
if (_stagingBuffer.Handle != 0)
{
_context.Vk.DestroyBuffer(_context.Device, _stagingBuffer, null);
_context.Vk.FreeMemory(_context.Device, _stagingMemory, null);
}
_stagingSize = size;
_stagingBuffer = CreateBuffer(size, BufferUsageFlags.TransferDstBit);
var requirements = GetMemoryRequirements(_stagingBuffer);
_stagingMemory = AllocateMemory(requirements, MemoryPropertyFlags.HostVisibleBit | MemoryPropertyFlags.HostCoherentBit);
var result = _context.Vk.BindBufferMemory(_context.Device, _stagingBuffer, _stagingMemory, 0);
if (result != Result.Success)
throw new InvalidOperationException($"vkBindBufferMemory failed: {result}");
}
private Silk.NET.Vulkan.Buffer CreateBuffer(ulong size, BufferUsageFlags usage)
{
var createInfo = new BufferCreateInfo
{
SType = StructureType.BufferCreateInfo,
Size = size,
Usage = usage,
SharingMode = SharingMode.Exclusive
};
Silk.NET.Vulkan.Buffer buffer;
var result = _context.Vk.CreateBuffer(_context.Device, &createInfo, null, &buffer);
if (result != Result.Success)
throw new InvalidOperationException($"vkCreateBuffer failed: {result}");
return buffer;
}
private MemoryRequirements GetMemoryRequirements(Silk.NET.Vulkan.Buffer buffer)
{
MemoryRequirements requirements;
_context.Vk.GetBufferMemoryRequirements(_context.Device, buffer, &requirements);
return requirements;
}
private DeviceMemory AllocateMemory(MemoryRequirements requirements, MemoryPropertyFlags properties)
{
var memoryTypeIndex = FindMemoryType(requirements.MemoryTypeBits, properties);
var allocInfo = new MemoryAllocateInfo
{
SType = StructureType.MemoryAllocateInfo,
AllocationSize = requirements.Size,
MemoryTypeIndex = memoryTypeIndex
};
DeviceMemory memory;
var result = _context.Vk.AllocateMemory(_context.Device, &allocInfo, null, &memory);
if (result != Result.Success)
throw new InvalidOperationException($"vkAllocateMemory failed: {result}");
return memory;
}
private uint FindMemoryType(uint typeFilter, MemoryPropertyFlags properties)
{
PhysicalDeviceMemoryProperties memoryProperties;
_context.Vk.GetPhysicalDeviceMemoryProperties(_context.PhysicalDevice, &memoryProperties);
for (var i = 0; i < memoryProperties.MemoryTypeCount; i++)
{
if ((typeFilter & (1u << i)) != 0 &&
(memoryProperties.MemoryTypes[i].PropertyFlags & properties) == properties)
{
return (uint)i;
}
}
throw new InvalidOperationException("Failed to find suitable memory type.");
}
private static uint GetPixelSize(Format format)
{
return format switch
{
Format.B8G8R8A8Unorm or Format.B8G8R8A8Srgb or Format.R8G8B8A8Unorm or Format.R8G8B8A8Srgb => 4,
_ => throw new NotSupportedException($"Format {format} is not supported for screenshots.")
};
}
public void Dispose()
{
if (_stagingBuffer.Handle != 0)
{
_context.Vk.DeviceWaitIdle(_context.Device);
_context.Vk.DestroyBuffer(_context.Device, _stagingBuffer, null);
_context.Vk.FreeMemory(_context.Device, _stagingMemory, null);
}
}
}
+3
View File
@@ -30,6 +30,9 @@ public sealed unsafe class Swapchain : IDisposable
public Extent2D Extent => _extent; public Extent2D Extent => _extent;
public SwapchainKHR Handle => _swapchain; public SwapchainKHR Handle => _swapchain;
public uint ImageCount => (uint)_images.Length; public uint ImageCount => (uint)_images.Length;
public Format SurfaceFormat => _surfaceFormat.Format;
public Image GetImage(uint index) => _images[index];
public Swapchain(VulkanContext context) public Swapchain(VulkanContext context)
{ {