I have been going through the learning curve of reverse engineering. And since you fine folks have been so helpful, I wanted to share how I went through the process of finding where to hook direct3d's EndScene.
This is likely only interesting to n00bs (and people who want to poke fun
).
(btw, minor note that injecting threads into wow.exe and letting them run willy nilly is a recipe for disaster. doing my work in endscene made things so much better).
This is actually really straight forward, but being a total n00b... it took me a while get wrap my head around it.
Fundamentally, the only way an app does direct3d is to create the initial direct3d pointer via the api Direct3DCreate9. This returns a IDirect3D9 pointer and from there apps call IDirect3D9::CreateDevice, and it is that device pointer we are interested in (EndScene() hangs off it).
So, step 1... pull up ida and search for Direct3DCreate9. bingo, bongo, bungo. You quickly find a routine that takes 2 params... the hModule for d3d9.dll and the device pointer. So, we ask ourselves who calls this bad boy... and where do they store that pointer.
So, I look up the xrefs, there are 4 or so callers, and I get lost, and cry. Hey, I'm a n00b what did you expect?
Okay, take a step back.. what do I actually want? not the IDirect3D9 pointer, I want the IDirect3DDevice9 ... that is returned from IDirect3D9::CreateDevice.... how the hells do I find that?
Well, I could debug wow... and set a breakpoint, and see who is calling it. That sounds fun. So, I actually write some test code that calls these functions...
My code, with disasm:
Code:
IDirect3D9 *pD3D = Direct3DCreate9(D3D_SDK_VERSION);
0040185A push 20h
0040185C call Direct3DCreate9 (4031C0h)
00401861 mov dword ptr [pD3D],eax
pD3D->CreateDevice(D3DADAPTER_DEFAULT, D3DDEVTYPE_HAL, NULL, 3, NULL, NULL);
00401864 push 0
00401866 push 0
00401868 push 3
0040186A push 0
0040186C push 1
0040186E push 0
00401870 mov eax,dword ptr [pD3D]
00401873 mov ecx,dword ptr [eax]
00401875 mov edx,dword ptr [pD3D]
00401878 push edx
00401879 mov eax,dword ptr [ecx+40h]
0040187C call eax
Well, okay, we can see that 0x40 off from start of vtable is the pointer to CreateDevice. cool'ish... lets just run it and step into the "call eax"
Code:
6FD3B60A mov edi,edi
The only thing interesting here is the address -- 6FD3B60A. If I dig around a bit, my debugger tells me where d3d9 is actually loaded:
Code:
d3d9.dll 6FCF0000-6FEAA000
And 6FD3B60A minus 6FCF0000 = 4B60A for our offset from start of module.
So... wherever wow.exe loads d3d9.dll... 0x4b60a from there is where we want to set our breakpoint. When I did a test run... the module got loaded at the same exact address. Neat. (always that way? dunno).
so, break point at 6FD3B60A, and let wow.exe rip.
I am using windbg, here is the stack after we hit the breakpoint:
Code:
0:000> k
*** ERROR: Module load completed but symbols could not be loaded for Wow.exe
ChildEBP RetAddr
0019fabc 005de9f5 d3d9!CEnum::CreateDevice
WARNING: Stack unwind information not available. Following frames may be wrong.
0019fb68 005dfb2a Wow+0x1de9f5
0019fbc4 005d1ad7 Wow+0x1dfb2a
0019fbd0 005dfdcb Wow+0x1d1ad7
...
Okay, looks like we will be returning to 005de9f5 when we return from CreateDevice. Let's pull that up in IDA.
Code:
.text:005DE9D0 lea edi, [esi+3860h]
.text:005DE9D6 push edi ; IDirect3DDevice9 ** ppReturnedDeviceInterface
.text:005DE9D7 lea edx, [ebp+Dst]
.text:005DE9DA push edx ; D3DPRESENT_PARAMETERS * pPresentationParameters,
.text:005DE9DB mov edx, [esi+384Ch]
.text:005DE9E1 sbb ebx, ebx
.text:005DE9E3 and ebx, 30h
.text:005DE9E6 add ebx, 22h
.text:005DE9E9 push ebx ; DWORD BehaviorFlags,
.text:005DE9EA push edx ; HWND hFocusWindow,
.text:005DE9EB push 1 ; D3DDEVTYPE DeviceType (HAL)
.text:005DE9ED push 0 ; UINT Adapter
.text:005DE9EF push eax
.text:005DE9F0 mov eax, [ecx+40h]
.text:005DE9F3 call eax
w00t! 
I added param commenting, but this is clearly goodness. The pointer is stored at 0x384c away from esi, ... a bit further back in the code you will see that esi just came from ecx (this pointer), which I call just "CGame". Same pointer that is referenced some 2860 times in wow.exe
So, something like:
Code:
struct CGame
{
char WhoKnows(0x3860);
IDirect3DDevice9 *pDevice;
};
And you are pretty well in business. (from there, I just detour vtable[42] to "MyEndScene").
-Silly