[Question] D3D12 Present Hook Failing on Window Resize

[CodeBytes]

Hello,

I'm attempting to print a simple FPS counter to World of Warcraft from within D3D12's Present() function. I'm not too familiar with DirectX programming, so I'm using the code in this repository to help get me started. The DLL injects fine, and loads up the sample internal GUI without issue; however, when I resize the window it appears that the wow client resets its window somehow and the DLL ceases to function.

Allow me to elaborate on that last part. I wrote a basic app that launches multiple instances of World of Warcraft, resizes the windows (strips the borders and title bars too), and renames the windows. When I trigger a window swap, i.e. cause another window to become the main, full-sized, and focused window, the World of Warcraft client (only with the d3d dll injected) resets the window states of all windows. They default to their config.wtf settings and the window is renamed back to World of Warcraft. My DLL is still injected into the client, but the FPS display is no longer there.

I'd love to be able to debug my DLL but that's impossible (for me at least) with World of Warcraft's anti-debugger feature. So, instead, I compiled and attached my dll to this sample d3d12 application. To my surprise, the window swap did not cause my dll to fail. It seems to be something World of Warcraft specific. So, without the ability to debug I had to resort to the worst possible way of debugging a dll... the windows MessageBox.

After many hours, I discovered that if I comment out the following block of code, the window swap works without the windows resetting to their default states:

https://github.com/Sh0ckFR/Universal-Dear-ImGui-Hook/blob/master/d3d12hook.cpp#L94

for (size_t i = 0; i < buffersCounts; i++) {
	ID3D12Resource* pBackBuffer = nullptr;

	frameContext[i].main_render_target_descriptor = rtvHandle;

	// Commenting out this line breaks the GUI but prevents the windows
	// from resetting after a resize occurs
	pSwapChain->GetBuffer(i, IID_PPV_ARGS(&pBackBuffer));

	d3d12Device->CreateRenderTargetView(pBackBuffer, nullptr, rtvHandle);
	frameContext[i].main_render_target_resource = pBackBuffer;
	rtvHandle.ptr += rtvDescriptorSize;
}

In fact, the only thing I need to comment out is the IDXGISwapChain::GetBuffer call. Even simply calling pSwapChain->GetBuffer(0, IID_PPV_ARGS(&pBackBuffer)); from anywhere in the code will cause the windows to reset and the dll to fail after a window resize occurs.

Of course, we need to call IDXGISwapChain::GetBuffer so we can get the render target resource otherwise the GUI will not show at all.

https://github.com/Sh0ckFR/Universal-Dear-ImGui-Hook/blob/master/d3d12hook.cpp#L127

FrameContext& currentFrameContext = frameContext[pSwapChain->GetCurrentBackBufferIndex()];
currentFrameContext.commandAllocator->Reset();

D3D12_RESOURCE_BARRIER barrier;
barrier.Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION;
barrier.Flags = D3D12_RESOURCE_BARRIER_FLAG_NONE;
barrier.Transition.pResource = currentFrameContext.main_render_target_resource;  <<<<<<<<<<<<<<<<< Used here
barrier.Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES;
barrier.Transition.StateBefore = D3D12_RESOURCE_STATE_PRESENT;
barrier.Transition.StateAfter = D3D12_RESOURCE_STATE_RENDER_TARGET;

d3d12CommandList->Reset(currentFrameContext.commandAllocator, nullptr);
d3d12CommandList->ResourceBarrier(1, &barrier);
d3d12CommandList->OMSetRenderTargets(1, &currentFrameContext.main_render_target_descriptor, FALSE, nullptr);
d3d12CommandList->SetDescriptorHeaps(1, &d3d12DescriptorHeapImGuiRender);

ImGui::Render();
ImGui_ImplDX12_RenderDrawData(ImGui::GetDrawData(), d3d12CommandList);

barrier.Transition.StateBefore = D3D12_RESOURCE_STATE_RENDER_TARGET;
barrier.Transition.StateAfter = D3D12_RESOURCE_STATE_PRESENT;

d3d12CommandList->ResourceBarrier(1, &barrier);
d3d12CommandList->Close();

d3d12CommandQueue->ExecuteCommandLists(1, reinterpret_cast<ID3D12CommandList* const*>(&d3d12CommandList));

I found this interesting article by Microsoft which appeared to be relevant, but releasing all outstanding references to the swap chain's buffers using a WndProc callback as suggested did not solve the issue.

Does anyone have any ideas on how to solve this problem?

Thank you

[Icesythe7]

Hook d3d12's reset function and reset the state, while i haven't messed with d3d12 much here is how I do it in d3d9 which should be relatively the same concept in d3d12.

IReleaseD3dResourcesMutex.lock();
ImGui_ImplDX9_Shutdown();
ImGui_ImplWin32_Shutdown();
SetWindowLongPtr(CHandle, GWLP_WNDPROC, OldWndProc); //reset to original wndproc if hooked (rehook in present/endscene)
OldWndProc = 0;
ImGuiInitialized = false; //reset so that when (present/endscene) is called again imgui re-inits
IReleaseD3dResourcesMutex.unlock();
return Od9Reset(device, ukn);

[CodeBytes]

Thank you for your suggestion, Icesythe7. Unfortunately, this results in the same behavior as before. When I resize an injected window, the window resets to its default config.wtf settings. I also tried your suggestion with other methods (eg. ResizeBuffers()) but no cigar.

I was able to rule out Kiero and Dear ImGui as possible culprits, however. I'm going to start looking into open source fraps copycats to see how they go about this.

[Master674]

The GetBuffer call increments the reference count on the ID3D12Resource object so instead of storing the object in a raw pointer you should use CComPtr. Once the pBackBuffer object goes out of scope that way, the resource is released and the swapchain can properly reset

[CodeBytes]

Thank you, Master674. Unfortunately, the result is the same. The windows reset to their config.wtf settings.

            CComPtr<ID3D12Resource> pBackBuffer = nullptr;
            pSwapChain->GetBuffer(i, IID_PPV_ARGS(&pBackBuffer));

I did find something interesting when looking into OBS Studio. This bug report is precisely what I'm experiencing here, World of Warcraft and all. I hooked into D3D12's Release method (vtable offset 2); however, I'm experiencing an issue where any code in the function body causes an access violation exception. Might be a problem with minhook, not sure yet. Once I fix that I should be able to try the solution in the bug report.

[CodeBytes]

I was able to fix the crash in my IUnknown::Release() hook by calling the original function before running my logic, and then returning the result of the original function after my logic is processed. Unfortunately, the issue remains.

I am now using CComPtr for all D3D related pointers. Everything runs smooth on first load. I get my FPS reading and I can run my FPS limiting code. As soon as I resize the window, I lose the FPS display and the window resets itself back to the config.wtf settings.

When I call pSwapChain->GetDevice(__uuidof(ID3D12Device), (void**)&m_d3d12Device) after a resize, I get the com error "No such interface supported". I'm not sure what that means in this context yet, so I still have a bit of research ahead of me.

[CodeBytes]

I was able to fix the issue. Once a window resize occurs, the laws of physics are changed... metaphorically speaking of course. For example, using the IID_PPV_ARGS macro causes the object calling the referencing method to become invalid. For example:

// m_d3d12Device is valid

m_d3d12Device->CreateCommandAllocator(D3D12_COMMAND_LIST_TYPE_DIRECT, IID_PPV_ARGS(&m_d3d12Allocator));

// m_d3d12Device becomes invalid

Keep in mind, this only happens after a window resize. To prevent the object from becoming invalid, simply avoid using the IID_PPV_ARGS macro.

// m_d3d12Device is valid

m_d3d12Device->CreateCommandAllocator(D3D12_COMMAND_LIST_TYPE_DIRECT, __uuidof(ID3D12CommandAllocator), (void**)&m_d3d12Allocator);

// m_d3d12Device remains valid

Each time an object becomes invalid (including the original IDXGISwapChain3 passed to Present), World of Warcraft's recovery code kicks in and the window resets to its config.wtf settings. So you're going to want to implement error handling for every D3D method with an HRESULT return value. Since I need D3D to draw something on the screen, I had to resort to the window's message box to display any errors. In the future I'll create a named pipe to send log messages back to my external application.

Then there is the issue of timing your D3D object releases. I did this in an IUnknown::Release() hook. Be careful though, if you don't properly lock your release logic, it will get triggered tens of times causing your objects to become invalid and, once again, resulting in your window resetting to its config.wtf settings. I'm referring to condition locks of course. You'll also need to add a boolean toggle to check if the WM_SIZE message was processed, as this is the right time to trigger your release logic. Thanks to Microsoft for the article I referenced in my first post.

My release logic triggered a timed event which placed a short delay in the present hook before attempting to re-initialize all my variables. You'll have to reset Dear ImGui before re-initializing it, and also release your input hook. Also, as the aforementioned Microsoft article states, you will definitely want to release all outstanding references to the swap chain's buffers, otherwise--you guessed it... the window will reset to its original config.wtf settings.

This includes the frame contexts.

    for (uint32_t i = 0; i < m_bufferCounts; ++i)
    {
        if (FrameContext* context = GetFrameContext(i))
        {
            context->MainRenderTargetResource.Release();
            context->MainRenderTargetResource = nullptr;
            context = nullptr;
        }
    }

    m_frameContextMap.clear();

Of course I refactored the code, so the above obviously wont work with the original package and I'd expect you to also refactor. My frame contexts are stored in an unordered_map and the ID3D12CommandAllocator is not a part of that map.

There is still one frustrating caveat to all this. Since my application runs multiple instances of World of Warcraft, it's important to pack the windows as close to one another as possible. To do that I remove the siezebox and caption. However, removing the sizebox will cause all of this to fail when I programmatically change the window's size and position, and yes I'm going to say it again, once more causing the window to reset to its original config.wtf settings.

This is acceptable--albeit annoying--for the time being.

[34D]

I hook WndProc case WM_SIZE then CleanupRenderTarget, SetWindowLongPtr the Old Handle and set a bool flag.
when resize according to the flag to reacquire d3dDevice, SetWindowLongPtr, ImGui_Init, Release buffer in Detour DirectX Present

It seems to be no problem currently.

By the way, it is D3D11.