EndScene

[nwg601]

I am going crazy over this.

I've had it working in the past (many patches ago) but coming back to it can't get it to work.

typedef HRESULT (_stdcall *m_D3D_EndScene) (IDirect3DDevice9 *Device);
m_D3D_EndScene    m_ofEndScene;

Direct3D9__Device = 0xABF47C,            // 4.3.4_15595
Direct3D9__Device__OffsetA = 0x2800,  // 4.3.4_15595
Direct3D9__Device__OffsetB = 0xA8,    // 4.3.4_15595

HRESULT _stdcall MyEndScene(IDirect3DDevice9 *Device)
{
    return 0;
}
void applyHook(DWORD baseAddress)
{
    DWORD D3D9_Device;
    D3D9_Device = readMem<DWORD>(baseAddress + Direct3D9__Device);
    D3D9_Device = readMem<DWORD>(D3D9_Device + Direct3D9__Device__OffsetA);
    D3D9_Device = readMem<DWORD>(D3D9_Device);
    D3D9_Device = readMem<DWORD>(D3D9_Device + Direct3D9__Device__OffsetB);
    m_ofEndScene = (m_D3D_EndScene)DetourFunction((PBYTE)D3D9_Device, (PBYTE)MyEndScene);
}

I would expect a black screen (as I'm not calling original function from mine, but am instead getting a crash, any help would be appreciated.

EDIT :
I have obtained the same pointer from creating a new window, obtaining the device and using the VTable, so I'm fairly confident my pointer is good. I've also tried using ms detours 1.5 and 3 with no success.

So is it something wrong with my function?

HRESULT _stdcall MyEndScene(IDirect3DDevice9* Device)

If I understand correctly it's really a thiscall, but a thiscall works by passing 'this' on ecx, where stdcall passes the first param on ecx, so that should work?

[Threk]

Why don't you just use a pattern and the VTable ?

[nwg601]

I am investigating that at the moment, however even if I get that working I think I won't rest until I understand why this isn't working.

[migtron]

HRESULT _stdcall MyEndScene(IDirect3DDevice9* Device)

If I understand correctly it's really a thiscall, but a thiscall works by passing 'this' on ecx, where stdcall passes the first param on ecx, so that should work?

Where did you get the idea that stdcall would do anything with ECX? All arguments are passed on the stack and the callee cleans it. This means that your function is messing up the stack, as EndScene does not take any arguments apart from the pointer in ECX. It is rather odd that this supposedly worked beforehand.

[nwg601]

Where did you get the idea that stdcall would do anything with ECX? All arguments are passed on the stack and the callee cleans it. This means that your function is messing up the stack, as EndScene does not take any arguments apart from the pointer in ECX. It is rather odd that this supposedly worked beforehand.

You're right about stdcall, I'd pulled that from memory and hadn't double checked. However, after doing more research it looks like EndScene is actually stdcall, and that fits because all of the examples I've seen have been stdcall or winapi (typedef stdcall?).

[migtron]

You're right about stdcall, I'd pulled that from memory and hadn't double checked. However, after doing more research it looks like EndScene is actually stdcall, and that fits because all of the examples I've seen have been stdcall or winapi (typedef stdcall?).

Ah yes, I forgot that all those D3D "classes" are not real classes, but COM interfaces which are actually more like structs with function pointers in them using stdcall as the standard convention. In that case, the function definition should be correct.

[nwg601]

Yeah, that puts me back to square one. I have verified my pointer three different ways, I've used msdetours 1.5 and 3, I've written my own detour code and it still doesn't work. I'm thinking it might be something about my platform as I'm sure this should work, could it be something to do with using VS2012?

[Bananenbrot]

Did you check memory page protection flags? I don't know if Detours takes care of that for you...

[nwg601]

I haven't dealt with DEP before, if that's what you mean.

Is it silly of me to think I could do something like this:
VirtualAllocEx(procH,MyEndScene,t_size,MEM_COMMIT,PAGE_EXECUTE_READWRITE);

I'm really guessing at that, if that is a good path, how would I know what size to use?

Edit :
ollydbg seems to suggest that my dll isn't marked for execution, not sure how I would fix this.

[Bananenbrot]

try VirtualProtect function ... and be sure to reset the flags to its old state when you are done.

[nwg601]

Thanks, I appreciate your suggestions.

I'm currently using detours 1.5, and I get this
imgur: the simple image sharer

I'm really struggling to make sense of the error, why is the instruction at that address referencing itself?

VirtualProtect seems to succeed, returns non zero, but I can run it on my ES and original ES with no luck.

[Bananenbrot]

how about some debugging? do you know what is at the crashing memory location? print out pointers of EndScene, you hook, etc.
print out the first 6 bytes of your hooked function. and consider using a homebrewn detour solution, Detours is overkill and hard to debug if you don't know about the internals.

[_Mike]

Thanks, I appreciate your suggestions.

I'm currently using detours 1.5, and I get this
imgur: the simple image sharer

I'm really struggling to make sense of the error, why is the instruction at that address referencing itself?

VirtualProtect seems to succeed, returns non zero, but I can run it on my ES and original ES with no luck.

Re the referencing itself part, it means that the cpu isn't allowed to execute that instruction because of page protection flags. You need at minimum read access (the cpu needs to be able to read the instruction to be able to run it, x86 does not support execute-only pages), and if your cpu supports hardware DEP, you also need execute permission.
Like Bananenbrot said, use a debugger. (a source level one). Single-step through your hook setup code and verify all the variables for each step. Breakpoint EndScene and single-step through the detour and make sure it jumps to the right place.

[nwg601]

Oh dear, I feel very silly.

Thanks for all your help. The problem (and reason this worked before) is the new injector/reloader I wrote a few days ago, it automatically unloads the dll where MyEndScene resides after it's finished it's work.

I had tried to debug this in olly, which was showing the dll in the modules list so I didn't think about this (albeit with empty address space...)