Trouble hooking function with Detours

[Miivers]

I am having some trouble with hooking a function. I think I have figured out whats going wrong but I am not sure how I can resolve my problem without writing inline assembly. I am sorry for the wall of text but it is here for explaining my problem. I am using Microsoft detours for doing my hooks. My current hooking code is(simplified):

int __fastcall removeObject(int objectToRemove, char a2)
{
	int returnVal = thisPtr->rmObj(objectToRemove, a2);
	printf("Object: %#08x \n", returnVal);
	return returnVal;
}

typedef int(__fastcall *RemoveObject)(int objectToRemove, char a2);
DetourManager::RemoveObject rmObj = (RemoveObject)(void*)0x05DD500; //Function to hook

DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach(&(PVOID&)rmObj, removeObject);
DetourTransactionCommit();

Detours adds a "call __RTC_CheckEsp (0F863D0Fh)" after we call the hooked function and reports if something went wrong. I am getting the error message: "Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.". I am getting a access violation if I ignore this message and continue executing.


The function I am hooking is called from 0x004678FE as shown below where arguments are passed in EDX and ECX:

.text:004678F8 ; ---------------------------------------------------------------------------
.text:004678F8
.text:004678F8 loc_4678F8:                             ; CODE XREF: sub_467800+F1j
.text:004678F8                                         ; DATA XREF: .text:off_467AC8o
.text:004678F8 mov     edx, [ebx]                      ; jumptable 004678F1 cases 0,6,24,64
.text:004678FA mov     ecx, ebx
.text:004678FC push    0
.text:004678FE call    dword ptr [edx]					; Calling remove object
.text:00467900 jmp     short loc_46790A                ; jumptable 004678F1 default case
.text:00467902 ; ---------------------------------------------------------------------------

The function it self is shown below where Ida shows it as __thiscall but I am assuming it should be __fastcall(I am fairly new to reversing so please correct me if I am wrong here):

.text:005DD500 ; int __thiscall RemoveObject(int objectToRemove, char a2)
.text:005DD500 RemoveObject proc near                  ; DATA XREF: .rdata:off_80AF78o
.text:005DD500
.text:005DD500 arg_0= byte ptr  8
.text:005DD500
.text:005DD500 push    ebp
.text:005DD501 mov     ebp, esp
.text:005DD503 push    esi
.text:005DD504 mov     esi, ecx
.text:005DD506 call    ObjectManager_RmObject
.text:005DD50B test    [ebp+arg_0], 1
.text:005DD50F jz      short loc_5DD524
.text:005DD511 test    esi, esi
.text:005DD513 jz      short loc_5DD524
.text:005DD515 push    0                               ; int
.text:005DD517 push    0FFFFFFFFh                      ; ExitCode
.text:005DD519 push    offset aDelete                  ; "delete"
.text:005DD51E push    esi                             ; int
.text:005DD51F call    sub_646430
.text:005DD524
.text:005DD524 loc_5DD524:                             ; CODE XREF: RemoveObject+Fj
.text:005DD524                                         ; RemoveObject+13j
.text:005DD524 mov     eax, esi
.text:005DD526 pop     esi
.text:005DD527 pop     ebp
.text:005DD528 retn    4 ;ESP=0019FC58
.text:005DD528 RemoveObject endp
.text:005DD528
.text:005DD528 ; ---------------------------------------------------------------------------

My trampoline function "int __fastcall removeObject(int objectToRemove, char a2)" produces this assembly code:

int __fastcall removeObject_(int objectToRemove, char a2)
{
0F8732F0  push        ebp  
0F8732F1  mov         ebp,esp  
0F8732F3  sub         esp,0E4h  
0F8732F9  push        ebx  
0F8732FA  push        esi  
0F8732FB  push        edi  
0F8732FC  push        ecx  
0F8732FD  lea         edi,[ebp-0E4h]  
0F873303  mov         ecx,39h  
0F873308  mov         eax,0CCCCCCCCh  
0F87330D  rep stos    dword ptr es:[edi]  
0F87330F  pop         ecx  
0F873310  mov         byte ptr [a2],dl  
0F873313  mov         dword ptr [objectToRemove],ecx  
	int returnVal = thisPtr->rmObj(objectToRemove, a2);
0F873316  mov         esi,esp  
0F873318  mov         eax,dword ptr ds:[0F9EA054h]  
0F87331D  mov         dl,byte ptr [a2]  
0F873320  mov         ecx,dword ptr [objectToRemove]  
0F873323  mov         eax,dword ptr [eax]  
0F873325  call        eax  ;ESP = 0019FC5C
0F873327  cmp         esi,esp  ;ESI holds the value esb had before calling the function at 0x0F873325. ESI = 0019FC5C ESP = 0019FC60 (4 bytes difference)
0F873329  call        __RTC_CheckEsp (0F863D0Fh)  
0F87332E  mov         dword ptr [returnVal],eax

My problem is as I have commented at 0x0F873327 that ESP has grown by 4 bytes after returning from the hooked function. Everything seems to work as expected if I manually change ESP by subtracting 4 from it(from a debugger) before 0x0F873316.


To repeat the question. I am looking for a way to fix this without writing inline assembly.


Edit:
For thous interested. The function I am trying to hook is what I belive to be the function for removing objects from the objectManager for patch 1.12.1.

[Miivers]

I gave up trying to get the case a bow to work but I am still interested if someone is willing to come with a solution. I ended up with hooking another function instead.

If anyone is interested in functions for objectManager add/remove object(patch 1.12.1):
0x00613880 int __thiscall sub_613880(int this, int objectType) //where this is the object which is created
0x005FB5E0 int __thiscall sub_5FB5E0(int this) //where this is the object which is removed

[namreeb]

I tried to post before but OC was having problems.

The function you're calling I'm pretty sure is __thiscall, not __fastcall. Also, the function you're calling seems to be the CGPlayer_C destructor. I'm not sure that that is what you want.

[Master674]

I tried to post before but OC was having problems.

The function you're calling I'm pretty sure is __thiscall, not __fastcall. Also, the function you're calling seems to be the CGPlayer_C destructor. I'm not sure that that is what you want.

If I'm not mistaken, WoW objects are referenced in the object manager using an intrusive linked list so the destructor can actually remove the object from the object manager. But a destructor cannot have any arguments other than this (in this case __thiscall == __fastcall) so it cannot be the destructor.

The solution to your question is simple. Change your hook to have one dummy argument after the ECX parameter (for EDX).

int __fastcall someThiscallFunction(void* thisPtr, void* /* edx */, char);

[andy012345]

It's a thiscall with 1 argument (unknown size without seeing instruction opcode, so I'll assume your 8bit char is right), EDX is used to deref a pointer for the call.

[MaiN]

I've seen MSVC generate destructors that take a char flags argument. The flag indicates whether the object should free its own memory at the end of the destructor.
For inheritance hierarchies the most derived class passes 0 to its base class constructor so the memory is not freed before the destructor is done.