"Failed to create VkImage: -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY)" segmentation fault when resizing the window

[abian]

Using Vulkan backend with ./binaries/system/pyrogenesis -mod=mod -mod=public -mod=0ad-spirv -conf:rendererbackend:vulkan I get...

ERROR: Failed to create VkImage: -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY)
Assertion failed: "depthStencilAttachment->texture"
Location: Framebuffer.cpp:46 (Create)

... or...

ERROR: vkWaitForFences(device, 1, &fence.value, VK_TRUE, std::numeric_limits<uint64_t>::max()) returned -4 (VK_ERROR_DEVICE_LOST) instead of VK_SUCCESS
Assertion failed: "false && "vkWaitForFences(device, 1, &fence.value, VK_TRUE, std::numeric_limits<uint64_t>::max())""
Location: SubmitScheduler.cpp:145 (WaitUntilFree)

Steps to reproduce:

1. Using windowed mode, very high quality shadows, and Vulkan as the renderer backend, create new single player game, Acropolis Bay (2). Wait for the game to start.
2. Resize the window.

Stack trace:

ERROR: Failed to create VkImage: -2

Thread 1 "main" received signal SIGSEGV, Segmentation fault.
Renderer::Backend::Vulkan::CFramebuffer::Create (device=0x555557379dd0, name=0x555555cf6fdd "ShadowMapFramebuffer", colorAttachment=0x0, depthStencilAttachment=0x7fffffffd020) at ../../../source/renderer/backend/vulkan/Texture.h:47
47		uint32_t GetUsage() const override { return m_Usage; }
(gdb) bt
#0  Renderer::Backend::Vulkan::CFramebuffer::Create (device=0x555557379dd0, name=0x555555cf6fdd "ShadowMapFramebuffer", colorAttachment=0x0, depthStencilAttachment=0x7fffffffd020)
    at ../../../source/renderer/backend/vulkan/Texture.h:47
#1  0x00005555559fabb9 in Renderer::Backend::Vulkan::CDevice::CreateFramebuffer (this=<optimised out>, name=<optimised out>, colorAttachment=<optimised out>, depthStencilAttachment=<optimised out>)
    at ../../../source/renderer/backend/vulkan/Device.cpp:740
#2  0x00005555559b3997 in ShadowMapInternals::CreateTexture (this=0x555557c23000) at ../../../source/renderer/ShadowMap.cpp:590
#3  0x00005555559b46f5 in ShadowMap::SetupFrame (this=0x555557c21590, camera=..., lightdir=...) at ../../../source/renderer/ShadowMap.cpp:236
#4  0x00005555559a676e in CSceneRenderer::SetSceneCamera (this=<optimised out>, viewCamera=..., cullCamera=...) at ../../../source/renderer/SceneRenderer.cpp:958
#5  0x000055555592388d in CGameView::BeginFrame (this=0x5555588caca0) at ../../../source/graphics/GameView.cpp:231
#6  0x0000555555999979 in CRenderer::RenderFrameImpl (this=0x555557bcf270, renderGUI=<optimised out>, renderLogger=<optimised out>) at ../../../source/ps/Game.h:167
#7  0x000055555599ca30 in CRenderer::RenderFrame (this=0x555557bcf270, needsPresent=needsPresent@entry=true) at ../../../source/renderer/Renderer.cpp:461
#8  0x0000555555612fbf in Frame () at ../../../source/ps/Singleton.h:51
#9  RunGameOrAtlas (argv=...) at ../../../source/main.cpp:694
#10 0x00005555555fd72c in main (argc=5, argv=0x7fffffffdee8) at ../../../source/main.cpp:752

[Vladislav Belov]

-2 means VK_ERROR_OUT_OF_DEVICE_MEMORY. So it seems that the driver or GPU ran out of available memory. Generally we can't survive such error, only some places can do that.

Could you attach the crash stack? To do that you need to run the game with GDB: gdb -ex run /path/to/pyrogenesis and after the fault enter bt command into the GDB console to print the stack (if it wasn't printed automatically).

[abian]

Thanks for the pointers!

ERROR: Failed to create VkImage: -2

Thread 1 "main" received signal SIGSEGV, Segmentation fault.
Renderer::Backend::Vulkan::CFramebuffer::Create (device=0x555557379dd0, name=0x555555cf6fdd "ShadowMapFramebuffer", colorAttachment=0x0, depthStencilAttachment=0x7fffffffd020) at ../../../source/renderer/backend/vulkan/Texture.h:47
47		uint32_t GetUsage() const override { return m_Usage; }
(gdb) bt
#0  Renderer::Backend::Vulkan::CFramebuffer::Create (device=0x555557379dd0, name=0x555555cf6fdd "ShadowMapFramebuffer", colorAttachment=0x0, depthStencilAttachment=0x7fffffffd020)
    at ../../../source/renderer/backend/vulkan/Texture.h:47
#1  0x00005555559fabb9 in Renderer::Backend::Vulkan::CDevice::CreateFramebuffer (this=<optimised out>, name=<optimised out>, colorAttachment=<optimised out>, depthStencilAttachment=<optimised out>)
    at ../../../source/renderer/backend/vulkan/Device.cpp:740
#2  0x00005555559b3997 in ShadowMapInternals::CreateTexture (this=0x555557c23000) at ../../../source/renderer/ShadowMap.cpp:590
#3  0x00005555559b46f5 in ShadowMap::SetupFrame (this=0x555557c21590, camera=..., lightdir=...) at ../../../source/renderer/ShadowMap.cpp:236
#4  0x00005555559a676e in CSceneRenderer::SetSceneCamera (this=<optimised out>, viewCamera=..., cullCamera=...) at ../../../source/renderer/SceneRenderer.cpp:958
#5  0x000055555592388d in CGameView::BeginFrame (this=0x5555588caca0) at ../../../source/graphics/GameView.cpp:231
#6  0x0000555555999979 in CRenderer::RenderFrameImpl (this=0x555557bcf270, renderGUI=<optimised out>, renderLogger=<optimised out>) at ../../../source/ps/Game.h:167
#7  0x000055555599ca30 in CRenderer::RenderFrame (this=0x555557bcf270, needsPresent=needsPresent@entry=true) at ../../../source/renderer/Renderer.cpp:461
#8  0x0000555555612fbf in Frame () at ../../../source/ps/Singleton.h:51
#9  RunGameOrAtlas (argv=...) at ../../../source/main.cpp:694
#10 0x00005555555fd72c in main (argc=5, argv=0x7fffffffdee8) at ../../../source/main.cpp:752

[Vladislav Belov]

After r27839 there should be a readable error (not only its code).

[abian]

Yes, there is!

ERROR: Failed to create VkImage: -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY)
Assertion failed: "depthStencilAttachment->texture"
Location: Framebuffer.cpp:46 (Create)

Sometimes I get that, and sometimes I get -4 (VK_ERROR_DEVICE_LOST) instead...

ERROR: vkWaitForFences(device, 1, &fence.value, VK_TRUE, std::numeric_limits<uint64_t>::max()) returned -4 (VK_ERROR_DEVICE_LOST) instead of VK_SUCCESS
Assertion failed: "false && "vkWaitForFences(device, 1, &fence.value, VK_TRUE, std::numeric_limits<uint64_t>::max())""
Location: SubmitScheduler.cpp:145 (WaitUntilFree)

... but bt ends up displaying the same stack trace.

OpenGL works fine in these scenarios.

[Vladislav Belov]

Replying to abian:

ERROR: Failed to create VkImage: -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY)
Assertion failed: "depthStencilAttachment->texture"
Location: Framebuffer.cpp:46 (Create)

ERROR: vkWaitForFences(device, 1, &fence.value, VK_TRUE, std::numeric_limits<uint64_t>::max()) returned -4 (VK_ERROR_DEVICE_LOST) instead of VK_SUCCESS
Assertion failed: "false && "vkWaitForFences(device, 1, &fence.value, VK_TRUE, std::numeric_limits<uint64_t>::max())""
Location: SubmitScheduler.cpp:145 (WaitUntilFree)

So bt shows the same stack for both errors even if the assertion says different locations?

[abian]

When I get VK_ERROR_DEVICE_LOST, I see a debug window where I can specify Continue, Ignore, Debug, etc. As I can't write bt until I choose one of these options, I choose Debug, and then I get another one of those windows, I choose Debug again... and when I can finally write bt, I get the same stack trace. The debug windows only show memory addresses.

[Vladislav Belov]

Replying to abian:

When I get VK_ERROR_DEVICE_LOST, I see a debug window where I can specify Continue, Ignore, Debug, etc. As I can't write bt until I choose one of these options, I choose Debug, and then I get another one of those windows, I choose Debug again... and when I can finally write bt, I get the same stack trace. The debug windows only show memory addresses.

Could you attach userreport_hwdetect.txt and mainlog.html when VK_ERROR_DEVICE_LOST happens?

[abian]

Sure! Finally I was able to run bt earlier by choosing Exit instead of Debug or other options, so I'm attaching mainlog, userreport_hwdetect and stack trace when VK_ERROR_DEVICE_LOST happens.

[Vladislav Belov]

Replying to abian:

Sure! Finally I was able to run bt earlier by choosing Exit instead of Debug or other options, so I'm attaching mainlog, userreport_hwdetect and stack trace when VK_ERROR_DEVICE_LOST happens.

Thank you!

Could you install Vulkan SDK: ​https://vulkan.lunarg.com/sdk/home#linux ? It should allow to enable debug layers to tell more information. To enable them after the Vulkan SDK installation you need to add the following lines to your user.cfg (the last line is to disable false-positive warnings):

renderer.backend.debugcontext = "true"
renderer.backend.debugmessages = "true"
renderer.backend.debuglabels = "true"
renderer.backend.debugscopedlabels = "true"
renderer.backend.vulkan.disabledescriptorindexing = "true"

Those layers might tell additional information about errors but they also make the game slower. To disable them you need to remove those lines.

[abian]

Done!

[Vladislav Belov]

Replying to abian:

Done!

Could you run vkconfig command (it should open a window with different layers settings)? You would need to set Vulkan Layers Configurations to Validation and then in the Validation Settings section (right panel with checkboxes) enable all checkboxes in the Validation Areas excluding Synchronization, set GPU Base to GPU-Assisted, enable all checkboxes in the Best Practices and the run the game with it.

[abian]

Let's see.

[Vladislav Belov]

Replying to abian:

Let's see.

I've made a patch to check VK_ERROR_DEVICE_LOST: ​https://code.wildfiregames.com/D5135, could you test it?