EndScene hook does not work

[kingdeking]

Hello,

I am trying to apply an EndScene hook on WoW using a VMT hook (VMT hook is just replacing the pointer in the Virtual Method Table of D3D9 to point to your own function).
Here is my Code:

All pointers are valid. I debugged and looked up every pointer. Now what happens is that my own EndScene function (hkEndScene) is never called but the original function...

void InitHook()
{
	HMODULE hModule = NULL;
	while( !hModule )
	{
		hModule = GetModuleHandleA( "d3d9.dll" );
		Sleep( 100 );
	}

	DWORD* VTableStart = 0;
	DWORD VTable = dwFindPattern((DWORD)hModule, 0x128000, (PBYTE)"\xC7\x06\x00\x00\x00\x00\x89\x86\x00\x00\x00\x00\x89\x86", "xx????xx????xx");

	DWORD temp =  *(DWORD*)(VTable+2);
	VTableStart = (DWORD*)temp;
	o_pEndScene = (EndScene_t)VTableStart[42];

	Log *pLog = new Log("debug.log");
	char c[50];

	sprintf(c, "VTable pointer: %X", VTableStart);
	pLog->Write(c);

	sprintf(c, "VTable EndScene pointer: %X", VTableStart+42);
	pLog->Write(c);

	pLog->Write("Attempting to hook EndScene");

	sprintf(c, "Before: VTableStart[42]: %X", VTableStart[42]);
	pLog->Write(c);

	DWORD oldRights;
	pLog->Write("Unprotecting memory");

	VirtualProtect(&VTableStart[42], 4, PAGE_EXECUTE_READWRITE, &oldRights);
	VTableStart[42] = (DWORD)hkEndScene;
	VirtualProtect(&VTableStart[42], 4, oldRights, &oldRights);

	pLog->Write("Reprotecting memory");

	pLog->Write("EndScene hooked");

	sprintf(c, "o_pEndScene: %X", o_pEndScene);
	pLog->Write(c);

	sprintf(c, "hkEndScene: %X", hkEndScene);
	pLog->Write(c);

	sprintf(c, "After: VTableStart[42]: %X", VTableStart[42]);
	pLog->Write(c);
}

The log looks like this:

VTable pointer: 5F324E08

VTable EndScene pointer: 5F324EB0

Attempting to hook EndScene

Before: VTableStart[42]: 5F34279F

Unprotecting memory

Reprotecting memory

EndScene hooked

o_pEndScene: 5F34279F

hkEndScene: 6E611000

After: VTableStart[42]: 6E611000

And here is hkEndScene:

HRESULT __stdcall hkEndScene(LPDIRECT3DDEVICE9 pDevice)
{
	pLogg->Write("hkEndScene called");

	const D3DCOLOR txtPink = D3DCOLOR_ARGB(255, 255, 0, 255);
	DrawRect( pDevice, 10, 10, 200, 200, txtPink);

	return o_pEndScene(pDevice);
}

[suicidity]

Before: VTableStart[42]: 5F34279F

That doesn't look correct.

[Bananenbrot]

First, Before: VTableStart[42]: 5F34279F should have made you suspicious... functions are usually aligned (by at least 4 bytes), so that F indicates an error in your calculations.
On a side note, your C(++ ) is more than questionable... sprintf in C++? if you decide not to use stringstream or a stream overload in your log class (which is understandable... it's quite ugly compared to boosts format library or printf-like functions), then at leasat use sprintf_n... IIRC there was even a compiler warning for it

Edit: And I am too slow....

[kingdeking]

That doesn't look correct.

But look where I end up in Olly:



And its breaking all the time... What is this shit Might be a different function then?

edit: When I compare it to EndScence in IDA, its similar. So this must be EndScene guys. The funny thing: when I detour EndScene,
the hook gets called but only once and then wow freezes, no idea why.

edit2: I am injecting with Winject...

edit3: Okay, its not working in a different game either, so it must be my problem.

edit4: Okay, its not my problem, i tested different hook and pointers are fine. It must be a problem with applying the hook too late.

[_Mike]

But look where I end up in Olly:



And its breaking all the time... What is this shit Might be a different function then?

edit: When I compare it to EndScence in IDA, its similar. So this must be EndScene guys. The funny thing: when I detour EndScene,
the hook gets called but only once and then wow freezes, no idea why.

edit2: I am injecting with Winject...

edit3: Okay, its not working in a different game either, so it must be my problem.

edit4: Okay, its not my problem, i tested different hook and pointers are fine. It must be a problem with applying the hook too late.

You are changing the VMT of CD3DHal which has no effect on CD3DBase, which is the one you want.

.text:7542848B C7 06 C8 50 41 75         mov    dword ptr [esi], offset ??_7CD3DBase@@6B@ ; const CD3DBase::`vftable'

Create a pattern from that.

[kingdeking]

Thanks for all the replies! I finally got it working. The problem was what I already suspected: The hook has been applied too late. I wrote my own injector that will start WoW in suspended mode and resume WoW once the DLL is injected. What a stupid problem

[kingdeking]

You are changing the VMT of CD3DHal which has no effect on CD3DBase, which is the one you want.

.text:7542848B C7 06 C8 50 41 75         mov    dword ptr [esi], offset ??_7CD3DBase@@6B@ ; const CD3DBase::`vftable'

Create a pattern from that.

Thanks for your answer, but its working fine now. Btw, are you sure I am changing the VMT of CD3DHal. I found this pattern on the Internet...

[_Mike]

Yeah you had the right VMT. I got things mixed up. EndScene's code is in CD3DBase but the actual device is a CD3DHal object which inherits from CD3DBase. Sorry