GCC "thiscall" calling convention [Linux/Win32 MinGW]

[Sednogmah]

If you're trying to write an in-process bot using the GNU C or C++ Compiler, you'll run into a problem sooner or later: The GCC calling convention for member functions (thiscall) is different to MS VC++ which means that you can't just call member functions like CGPlayer::ClickToMove without a workaround.

First, the particular differences for thiscall are:

MS Visual C++:
* Arguments are pushed on the stack in reverse order.
* The object pointer is passed in ECX.
* The callee pops arguments when returning.
* Primitive data types, except floating point values, are returned in EAX or EAX:EDX depending on the size.
* float and double are returned in fp0, i.e. the first floating point register.
* Simple data structures with 8 bytes or less in size are returned in EAX:EDX.
* All classes and structures are returned in memory, regardless of size.
* When a return is made in memory the caller passes a pointer to the memory location as the first parameter (hidden). The callee populates the memory, and returns the pointer. The callee pops the hidden pointer together with the rest of the arguments.
* If the method takes variable number of arguments, i.e. is declared with the ... operator, then the calling convention instead becomes that of cdecl. The object pointer is pushed on the stack as the first argument instead of passed in ECX, and all arguments are popped from the stack by the caller when the function returns.

MinGW g++ / Win32:
* Arguments are pushed on the stack in reverse order.
* The object pointer is pushed on the stack as the first parameter.
* The caller pops arguments after return.
* Primitive data types, except floating point values, are returned in EAX or EAX:EDX depending on the size.
* float and double are returned in fp0, i.e. the first floating point register.
* Objects with 8 bytes or less in size are returned in EAX:EDX.
* Objects larger than 8 bytes are returned in memory.
* Classes that have a destructor are returned in memory regardless of size.
* When a return is made in memory the caller passes a pointer to the memory location as the first parameter (hidden). The callee populates the memory, and returns the pointer. The callee pops the hidden pointer from the stack when returning.
* Classes that have a destructor are always passed by reference, even if the parameter is defined to be by value.

GCC g++ / Linux:
* Arguments are pushed on the stack in reverse order.
* The object pointer is pushed on the stack as the first parameter.
* The caller pops arguments after return.
* Primitive data types, except floating point values, are returned in EAX or EAX:EDX depending on the size.
* float and double are returned in fp0, i.e. the first floating point register.
* All structures and classes are returned in memory regardless of complexity and size.
* When a return is made in memory the caller passes a pointer to the memory location as the first parameter (hidden). The callee populates the memory, and returns the pointer. The callee pops the hidden pointer from the stack when returning.
* Classes that have a destructor are always passed by reference, even if the parameter is defined to be by value.

I will use the ClickToMove function as an example for my workaround. The MS VC++ definition of ClickToMove is:

BOOL __thiscall CGPlayer_C__ClickToMove(
   WoWActivePlayer *this, CLICKTOMOVETYPE clickType,
   WGUID *interactGuid, WOWPOS *clickPos, float precision);

The first difference I'm compensating for is that MS VC++ code, like WoW, expects the callee to pop the arguments from the stack before it returns. This can be achieved by declaring the function with the attribute stdcall:

typedef bool (*fp_CGPlayer_C__ClickToMove) (
  uint32_t ctmtype, uint64_t* clickGUID,
  float* clickPos, float precision)
  __attribute__((stdcall));

Note that the function call is missing the instance pointer "WoWActivePlayer *this". As MS VC++ member functions expect that pointer to be in the "ecx" register, I'm setting it with inline assember before the function call.

asm("movl %0, %%ecx" : : "m" (activePlayer));
ClickToMove(ctmType, pClickGUID, pClickPos, 0);

Note that GCC uses AT&T and not Intel assembler syntax. (opcode src,dest instead of dst,src).

Sources:
* Calling conventions on the x86 platform
* Function Attributes - Using the GNU Compiler Collection (GCC)
* Extended Asm - Using the GNU Compiler Collection (GCC)

This works fine for me. Constructive criticism is welcome.

[eLaps]

+2rep
What about cdecl functions? They just work fine?

[flo8464]

+2rep
What about cdecl functions? They just work fine?

Yes. It is really horrible that there is no real standard for compilers how to treat c++-calls and especially name mangling.

[Sednogmah]

+2rep
What about cdecl functions? They just work fine?

Yes, normal cdecl functions are fine, as flo8464 already said. Maybe some moderator can change the thread's topic to: GCC "thiscall" calling convention [Linux/Win32 MinGW] to make it more clear.

[Apoc]

Yes, normal cdecl functions are fine, as flo8464 already said. Maybe some moderator can change the thread's topic to: GCC "thiscall" calling convention [Linux/Win32 MinGW] to make it more clear.

Done.

Next time, just PM me. I don't troll these threads as much as I used to. (I only skim thru posts now)

At any rate; very helpful info for those getting started. Although; not many windows developers (especially 'hack' writers) use GCC. Unless of course we're talking Wine, which in itself, has a small userbase. (A few thousand maybe? With only at most a few hundred [if that much] hacking on it.)

[pistos]

wandering on the internet I found this thread, and I have to add something.

in gcc 4.6.x, you can use the __thiscall keyword, and it'll be fine.

But, since it's very new and possibly buggy, there's another way, cleaner than inline assembly (which could not work if the compiler does weird things in the function epilog), to create a pseudo-thiscall function. Just declare your function as a __fastcall. This means that the first two arguments of a function are passed in ECX and EDX. You don't need the second argument, so you just pad the argument list of your function with a dummy argument in the second place.
The example in the first post would become:

BOOL __fastcall CGPlayer_C__ClickToMove(
   WoWActivePlayer *this, int, CLICKTOMOVETYPE clickType,
   WGUID *interactGuid, WOWPOS *clickPos, float precision);

notice that you can even omit the argument name.

[eLaps]

...

Indeed, that's clever. +rep

[Sednogmah]

Good stuff, pistos, both the new __thiscall and your __fastcall hack!

[Dmytry]

Massive cudos to pistos, saved my day. In some other API i'm now using a variation of his method:

#if defined(__MINGW32__)
#define EVIL_API_CALL __attribute__((fastcall)) // might also help to add returns_twice if i get some weird register clobbering issue
#define START_ARG_DECL int,
#define NO_ARG_DECL int
#define START_ARG 0,
#define NO_ARG 0
#else
#define EVIL_API_CALL
#define START_ARG_DECL
#define NO_ARG_DECL
#define START_ARG
#define NO_ARG
#endif
..
class ISomeInterface{
virtual EVIL_API_CALL bool GetSomething(START_ARG_DECL const char *name, int32 *data ) = 0;
}
...
Accessor()->GetSomething(START_ARG "blabla", &data);

this way my code works with msvc or on OS X where the api i am using uses GCC and it works perfect.
(can't tell what API it is because of NDA)
One massive gotcha: watch out for vtable ordering. MSVC groups AND re-orders overloaded virtual methods; i am not sure what ordering MSVC makes, either some sorting or reverse order. I wasted few hours believing that i had some problems with prologue/epilogue of my override of virtual function.

[Cypher]

Massive cudos to pistos, saved my day. In some other API i'm now using a variation of his method:

#if defined(__MINGW32__)
#define EVIL_API_CALL __attribute__((fastcall)) // might also help to add returns_twice if i get some weird register clobbering issue
#define START_ARG_DECL int,
#define NO_ARG_DECL int
#define START_ARG 0,
#define NO_ARG 0
#else
#define EVIL_API_CALL
#define START_ARG_DECL
#define NO_ARG_DECL
#define START_ARG
#define NO_ARG
#endif
..
class ISomeInterface{
virtual EVIL_API_CALL bool GetSomething(START_ARG_DECL const char *name, int32 *data ) = 0;
}
...
Accessor()->GetSomething(START_ARG "blabla", &data);

this way my code works with msvc or on OS X where the api i am using uses GCC and it works perfect.
(can't tell what API it is because of NDA)
One massive gotcha: watch out for vtable ordering. MSVC groups AND re-orders overloaded virtual methods; i am not sure what ordering MSVC makes, either some sorting or reverse order. I wasted few hours believing that i had some problems with prologue/epilogue of my override of virtual function.

Another thing to watch out for in MSVC is use of '__int64'.

I haven't confirmed it's still the case, but ages ago when I was writing an injected hack, I noticed that if you use __int64 as a data member of your class, MSVC will 'pad' the class in memory with an extra 4 bytes.

Assuming the object has a vtable pointer at offset 0x0, that will put your first data member at 0x8 rather than 0x4. That one cost me about an hour's worth of time and hair pulling. :P

[Dmytry]

Thanks, cypher, that's good to know!
I just found something probably quite useful:
Structure-Packing Pragmas - Using the GNU Compiler Collection (GCC)
and
Type Attributes - Using the GNU Compiler Collection (GCC)
(scroll down to "Two attributes are currently defined for i386 configurations: ms_struct and gcc_struct. ")
I wonder if there is compatible vtable ordering mode as well, in new GCC.
I'm using gcc 4.4.0 though and I cannot update because I'm quite close to deadline.

[caytchen]

Just don't bother defining virtual functions in your classes trying to replicate the exact class layout. This is just brain dead if you try it on GCC for a program compiled with MSVC, since you aren't guaranteed anything regarding the generated assembly code for either. This might be a viable tactic for MSVC compiled hacks, partly because WoW is not built with aggressive optimizations like PGO, but the comfort you gain is I think not worth it since you end up cluttering your class definitions with lots and lots of virtual filler functions.

[Cypher]

Just don't bother defining virtual functions in your classes trying to replicate the exact class layout. This is just brain dead if you try it on GCC for a program compiled with MSVC, since you aren't guaranteed anything regarding the generated assembly code for either. This might be a viable tactic for MSVC compiled hacks, partly because WoW is not built with aggressive optimizations like PGO, but the comfort you gain is I think not worth it since you end up cluttering your class definitions with lots and lots of virtual filler functions.

Technically you only end up cluttering your class DECLARATIONS. You can just abuse pure virtual functions to avoid providing a definition.*

Example:

class Foo
{
public:
	virtual void Padding00() = 0;
	virtual void Padding04() = 0;
	virtual char* GetName() = 0;
	virtual void Padding08() = 0;
	virtual float GetPitch() = 0;
};

Obviously you won't be able to instantiate the class, but when it comes to WoW you will never need to as you're always operating on pointers to existing objects.

*Inb4 some retard comes along with the whole HURR DURR A DEFINITION IS ALSO A DECLARATION crap that comes up on Usenet and mailing lists all the time thanks to trolls and unmoderated lists.

P.S. Sorry if that's what you meant, I know that you probably already knew this trick, but a surprising amount of people don't, so I figured it was best to get it out there regardless.

[Dmytry]

Just don't bother defining virtual functions in your classes trying to replicate the exact class layout. This is just brain dead if you try it on GCC for a program compiled with MSVC, since you aren't guaranteed anything regarding the generated assembly code for either. This might be a viable tactic for MSVC compiled hacks, partly because WoW is not built with aggressive optimizations like PGO, but the comfort you gain is I think not worth it since you end up cluttering your class definitions with lots and lots of virtual filler functions.

Works for me. Easily replicated MSVC vtable ordering, for pure virtual interfaces. The *only* thing I needed to do was to switch order of overloads when defined(__MINGW32__) .
Luckily, he API I am having to deal with uses abstract interfaces, works a bit like COM, does not have a lot of overloads, and puts those nearby (as well as using #pragma pack( ). Without overloading, vtable order is same (if it was not, COM from GCC would of been much more difficult than it is. COM prohibits overloads).
It all works, and it was a lot easier than alternative, which would of been writing a wrapper and compiling it in MSVC (using proper COM in the wrapper), which would of took way longer. I'm not actually hacking; I'm developing a game... I cannot change compiler easily because I support Linux and OS X as well and I will need networked play compatibility, which is so much easier when using same compiler (GCC). The API is for some rather minor feature.

And yes, you are guaranteed quite a lot. As long as you don't try to use STL classes over DLL boundary, your code will work across MSVC versions. Microsoft's class layout is very stable; the reason they have this stupid reordering of overloads is backwards compatibility. Microsoft is VERY serious about backwards compatibility. GCC is a free compiler and it's behaviour is well documented, so you are guaranteed *everything*. PGO, well if you are not guaranteed particular layout with PGO then it will break compatibility with MSVC just as well as it would with GCC.

Anyways there's the game where i'm using this stuff now (I'm having to interface to some API, the 'official' way, except the api is not nice to other compilers)

[amadmonk]

ZOMBIE! Just had to comment on this nearly year-old thread. As someone who used to work on the MSVC team, I should point out that both internal calling conventions and structure packing/layout are, technically, implementation-defined. So it's not that MSVC is "wrong" and GCC is "right" (or vice-versa). Just different.

Of course, this implementation-dependent thing may have changed since FWIK C++0x has done some work to define backwards "binary compatibility" w/C, but back when we were doing the MSVC compiler (circa VS2k3... yes, I'm old), the ONLY thing that guided us in this respect was the ANSI/ISO standards (which left the implementation up to the compiler writer) and the Intel ABI, which just gave hints and a few hard-coded specifications (specifically calling conventions).

<end defensive rant for a product that I no longer work around>