Question regarding DLL-export and problems with the stack

[schlumpf]

Why does

extern "C" __declspec(dllexport) bool ScanFake(int *a, int, int, int)
{
	*a = 0;
	return true;
}

compile to

sub proc near
arg_0= dword ptr  4

mov     eax, [esp+arg_0]
and     dword ptr [eax], 0
mov     al, 1
retn
sub endp

instead of

sub proc near
arg_0= dword ptr  4

push    ebp
mov     ebp, esp
mov     ecx, [ebp+arg_0]
mov	[ecx],0
mov     al, 1
pop     ebp
retn
sub endp

Any thoughts how I can get the compiler (VC++) to modify the stack so the four pushed ints will be off it? As soon as I add a __asm{ add esp, 16 }, it will the stack management. But that will lead to ****ing it up again. (Since it will pop too much then.

push    ebp
mov     ebp, esp
mov     eax, [ebp+arg_0]
and     dword ptr [eax], 0
add     esp, 10h
mov     al, 1
pop     ebp
retn

Well: Which parameters do I have to change to get the compiler to add push ebp etc.?

[corderoy]

I think the term you are looking for is "calling convention", as well as "__cdecl" and "__stdcall". Check those out at google or here
to get more info.

[schlumpf]

Okay, but why is it automatically changing as soon as I modify the stack by hand?

Thanks for giving me that hint. Forgot about it since I use a macro to do the type of the function.

[Cypher]

It's probably not bothering with a stack frame because the function is so tiny that it can be optimized out. The point of a stack frame is to make it easier to access both local vars and params. You don't have any local vars so it can just grab the params off ESP without backing it up and doing a sub for you local vars.

Argh afk. I'll finish explaining later.

[Cypher]

Right. First off, if you want to do any manual stack modifications you MUST use __declspec(naked) otherwise you'll **** up a whole range of things.

Also, the first function is output the way it is because of optimizations. It dereferences the pointer and uses an AND to zero it out. Then it just returns true. It doesn't bother doing and movs/subs because there's no need, the function is just too simple to make it worthwhile.

EDIT: Also, you don't need to change params to get a stack frame. Try adding about 10-20 bytes of local variabels and see how it changes. Optimization settings will make a difference too.

[schlumpf]

Well, I can understand optimization, but if I pass four arguments, why wont it pop them again? When will it add code to pop them? As soon, as I change the value of them or use them for something?

The problem was, that I was unable to change the call resulting in four pushs and no pops.

[kynox]

Did you not read corderoy's post when he directed you to calling conventions?

__cdecl requires the caller to fix the stack, __stdcall requires the callee to fix the stack.

[schlumpf]

As soon as I tried to fix the stack, it automatically changed to stdcall. This was my problem.

[Cypher]

That's because you're dumping inline ASM into a function that modifies the stack without ensuring that it's 'legal' to do so. Not only were you not marking the calling convention correctly, if you want to set up a custom stack frame you need to mark the function as naked.

I see what you meant now by the original function but as kynox said, you were probably using __cdecl, which is the default C++ calling convention for most major compilers. If you want to use __stdcall declare your function like this:

void __stdcal Foo(int P1, int P2, int P3, int P4)
{
// code goes here
}

Fairly simple stuff. Its all documented under the calling conventions section of MSDN or any decent reversing book.