Edit: Orignal post was bloated and vague, i had to rewrite it. changed/edited the questions while i was at it. luckrunsout's repsonse was based on the original post.
Im confused about d3d9 hooking and I would be glad if someone could clear some things up for me. Im working on the 2.4.3 client.
Ive made a working hook which can call ingame functions. Im using what I think is a IDirect3dDevice9 pointer which i found by setting a breakpoint and looking at the endscene caller in wow module (see code below).
Using that device I replace the pointer in vtable[42] and after my code has finished i jump to the real endscene. Im not injecting a dll but using zenlulz memorysharp library to write assembly into my allocated memory, its probably not optimal but seemed easier to grasp (one less step to figure out).
PHP Code:
//Calls endscene
Wow.exe+1AB230 - 56 - push esi
Wow.exe+1AB231 - 8B F1 - mov esi,ecx
Wow.exe+1AB233 - 83 BE F4390000 00 - cmp dword ptr [esi+000039F4],00
Wow.exe+1AB23A - 74 1B - je Wow.exe+1AB257
Wow.exe+1AB23C - 8B 86 64380000 - mov eax,[esi+00003864] //IDirect3dDevice9? + vtable offset?
Wow.exe+1AB242 - 8B 08 - mov ecx,[eax]
Wow.exe+1AB244 - 8B 91 A8000000 - mov edx,[ecx+000000A8] // (aha, a8: vtable endscene offset)
Wow.exe+1AB24A - 50 - push eax
Wow.exe+1AB24B - FF D2 - call edx
Wow.exe+1AB24D - C7 86 F4390000 00000000 - mov [esi+000039F4],00000000
Wow.exe+1AB257 - 5E - pop esi
Wow.exe+1AB258 - C3 - ret
Basically, Ive experimented and guessed quite a lot to get to this point which makes me uncertain of my solution.
questions:
1. Ive heard of vtable hooks but the endscene hooks ive seen write a jump inside the function to detour, im guessing its because you can reliably get the address on any machine but im concerned that its because my way is detected by anticheat. If you wanted to detect the hook checking if the pointer is valid shouldnt be difficult. Basically, is what im doing inadvisable or flat out dangerous from an anti cheat standpoint? (emu or retail warden)
2. Theres a post or two on the forums which mention that you shouldnt call lua_dostring from an endscene hook in retail anymore. Is that an issue on emuservers?