Rust: Calling Functions With Different Calling Conventions From Same File

The other day I had some fun with assembly and wanted to try to call assembly functions from Rust. Each compiler follows a certain calling convention, that specifies how parameters are passed between functions. The calling convention is part of an ABI, an application binary interface. There are two major calling conventions out there: System V ABI and “Microsoft/PE x64”. The latter has no real official name, at least I can’t find one in the Microsoft docs.

Each processor has a set of registers (on x86_64 for example rax, rcx, rdx, rsi, rdi) which are mainly used for transferring parameters. For example, on Microsoft Windows (or UEFI, which uses the same convention) the first two integer arguments are passed in rcx and rdx. On UNIX on the other hand, they get passed via rdi and rsi.

I wrote a simple add(u64, u64) -> u64 function in assembly, that takes two 64-bit integers and calculates the sum. There are two implementations, where one uses the System V ABI and the other the “Microsoft/PE x64” convention to access the parameters. From Rust this is really easy to manage, because we can tell the compiler what calling convention it should use. This results in a compiler output by Rust, where the parameters are passed the right way. But first let’s look at my assembly code.

# written in GAS (GNU Assembly)

.section .text


    # Adds two 64-bit numbers and returns the 64-bit result by using the
    # Microsoft calling convention used in the "PE" format on Windows
    # and UEFI.
    #
    # More info: https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-160
    #
    # First param:  RCX
    # Second param: RDX
    # Return:       RAX
    pe_abi__asm_add:
        mov     rax, rcx
        add     rax, rdx
        ret


    # See Rust code. Thin wrapper to demonstrate,
    # that UEFI indeed uses the calling convention used in the "PE" format.
    efi_abi__asm_add:
        call pe_abi__asm_add
        ret


    # See Rust code. Thin wrapper to demonstrate,
    # that Windows indeed uses the calling convention used in the "PE" format.
    win64_abi__asm_add:
        call pe_abi__asm_add
        ret

    # Adds two 64-bit numbers and returns the 64-bit result by using the
    # "System V ABI" calling convention present in Linux and MacOS.
    #
    # More info: https://www.uclibc.org/docs/psABI-x86_64.pdf
    #
    # First param:  RDI
    # Second param: RSI
    # Return:       RAX
    system_v_abi__asm_add:
        mov     rax, rdi
        add     rax, rsi
        ret

If we want to call this from Rust, we need some “extern” function definitions, which tells Rust about the calling convention. Down below you can see how this looks like:

// Behind the scenes, Windows uses "Microsoft/PE" calling convention.
//   https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-160
extern "win64" {
    fn win64_abi__asm_add(a: i64, b: i64) -> i64;
}

// Behind the scenes, UEFI uses "Microsoft/PE" calling convention.
//   https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-160
extern "efiapi" {
    fn efi_abi__asm_add(a: i64, b: i64) -> i64;
}

// Defaults to System V ABI (64 bit), i.e. the calling convention used on
// Linux or MacOS.
//   https://www.uclibc.org/docs/psABI-x86_64.pdf
extern "sysv64" {
    fn system_v_abi__asm_add(a: i64, b: i64) -> i64;
}

/// Calculates two numbers by using functions of two different calling conventions.
/// Furthermore, this example shows that "win64" and "efiapi" use indeed the "Microsoft/PE"
/// calling convention.
fn main() {
    println!("win64_abi__asm_add(2,7)   = {}", unsafe { win64_abi__asm_add(2, 7) });
    println!("efi_abi__asm_add(2,7)     = {}", unsafe { efi_abi__asm_add(2, 7) });
    println!("system_v_abi__asm_add(2,7)= {}", unsafe { system_v_abi__asm_add(2, 7) });
}

Site note: If we would write extern "C" the compiler would use the default C calling convention of the target (UNIX/Windows). Inside the Rust compiler source code we can find possible values for the extern "<calling convention/abi">. That’s all the magic. This tells the compiler in what registers things should be put. If you run it, it correctly calculates the sums. I have a demo project on Github where you can find the source code. If you want to see how it works to compile assembly functions along with Rust code, check out my previous blog post.

PS: This is nothing new but my past self (>5 years) would find this extreeeeemly cool.

Additional Resources

• Microsoft/PE calling convention: https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-160
• System V ABI: https://www.uclibc.org/docs/psABI-x86_64.pdf