Assembly - calling conventions - callers

Next up calling conventions, what does C do when it calls a function? Actually there are a few ways of calling functions, called calling conventions, there are choices about whether the caller or callee must restore registers, how argument and the return value are passed, how this is passed if it's a class method (not to mention virtual dispatch, but that's beyond the scope of this blog post). I will look at the following common conventions: cdecl (standard convention for C on x86), stdcall (used for the Win32 API), and fastcall (there are actually many varieties, but I'll look at the Microsoft version). There are many other conventions on different platforms and for different languages. In particular, there is thiscall, which is used by Microsoft for calling member functions, and a bunch of conventions for 64 bit processors, which favour registers, because there are more of them (cite: http://msdn.microsoft.com/en-us/library/984x0h58.aspx).

We'll use the call and ret instructions. In all cases, call saves the return address to the stack and ret pops it, so the return address is always on top of the stack when a call is made.

I'm going to continue to use the functions from last time, but now I'm going to add a calling convention to the signature and call them from assembly, rather than C.

Here are the new signatures:

void __cdecl copy(unsigned char* aSrc, unsigned char* aDest, unsigned int aLength);
void __stdcall fill(unsigned char* aDest, unsigned char aVal, unsigned int aLength);
void __fastcall fillWithCount(unsigned char* aDest, unsigned int aLength);

And here is the current calling code in C (there is some allocation before hand, and some printing after, but that is not important):

   fillWithCount(src, 256);
  fill(dest, 5, 256);
  copy(src, dest, 200);

cdecl
Arguments are passed on the stack in right to left order. Return value is in eax. eax, ecx, and edx must be preserved by the caller, other registers are preserved by the callee. The caller must clear the stack on return. So, the call to copy looks like:

//copy(src, dest, 200);
__asm {
  push 200
  push dest
  push src
  call copy
  add esp, 12
}

Which is actually not very interesting because there is no return value, nor registers to restore. The only thing of note is that we pop the three params off the stack by manually fiddling with the stack pointer.

stdcall

stdcall is similar to cdecl, but the callee tidies up the stack. So we don't need to pop anything from the stack.

//fill(dest, 5, 256);
__asm {
  push 256
  push 5
  push dest
  call fill
}

fastcall

fastcall is similar to stdcall in that the callee tidies up the stack, but two parameters are passed in ecx and edx, the rest on the stack like the other conventions. The return value is left in eax.

//fillWithCount(src, 256);
__asm {
  mov ecx, src
  mov edx, 256
  call fillWithCount
}