Practical Calling Conventions For Reverse Engineers (Patreon)

Calling Conventions

• Different conventions based on processor, OS, and language
• Describes how arguments are passed to functions
• Describes how values are returned from functions
• Describes if the caller or callee cleans the stack
• Responsible for function prologue and epilogue

Practical Calling Conventions

Windows x86 Calling Conventions

• All arguments are widened to 32 bits (DWORD)
• Return value is widened to 32 bits (DWORD)
• Return values up to 32 bits are returned in EAX
• Return values of 64 bit size (QWORD) are returned in EDX:EAX
• Return structures are returned by reference with a pointer in EAX
• Registers ESI, EDI, EBX, and EBP are restored (non-volatile)
• MSDN is your friend!

Calling Convention: __cdecl

• Default calling convention for C and C++
• Arguments are passed on the stack (pushed right-to-left)
• Caller is responsible for stack cleanup
• Supports vararg (variadic) functions

MSDN calling convention documentation 

More Calling Convention Tutorials 

• Calling Convention: x86 __stdcall 
• Calling Convention: x86 __fastcall 
• Calling Convention: x86 __thiscall 
• Calling Convention: IDA Pro__usercall 
• Calling Convention: x64 __fastcall 

Practice Examples

Attached to this post is a copy of the HelloWorld PE file from our tutorial. You are encouraged to open this sample in IDA and identify the function calling conventions as well work through the varargs access in the variadic function.