Author Topic: How to call win32 APIs in inlined at&t asm?  (Read 6130 times)

Offline reverser

  • Multiple posting newcomer
  • *
  • Posts: 17
How to call win32 APIs in inlined at&t asm?
« on: April 30, 2022, 12:32:20 pm »
hi all

Code
char * msg = "Hello, World!\n";
char * wMsg = "Content of the window..";
char * wCaption = "Window title";

int main (){
  std::cout << "before call" << "\n";

   asm(
"movl _msg,%eax\n\t"
"pushl %eax\n\t"
"calll _printf\n\t"
"popl %eax\n\t"
"movl $0, %eax\n\t"
"movl _wCaption, %ebx\n\t"
"movl _wMsg, %ecx\n\t"
"movl $0, %edx\n\t"
"pushl %eax\n\t"
"pushl %ebx\n\t"
"pushl %ecx\n\t"
"pushl %edx\n\t"
"calll MessageBoxA\n\t"
"popl %edx\n\t"
"popl %ecx\n\t"
"popl %ebx\n\t"
"popl %eax\n\t"
      );

    std::cout << "after call" << "\n";
}

when i want to compile, i get:
undefined reference to `MessageBoxA'

can someone please tell me what's wrong?
« Last Edit: April 30, 2022, 12:34:09 pm by reverser »

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6025
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: How to call win32 APIs in inlined at&t asm?
« Reply #1 on: April 30, 2022, 04:53:39 pm »
Just a guess: do you link the correct library?
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline reverser

  • Multiple posting newcomer
  • *
  • Posts: 17
Re: How to call win32 APIs in inlined at&t asm?
« Reply #2 on: May 01, 2022, 05:54:08 am »
Just a guess: do you link the correct library?
i don't think so
can you please tell me how? you have no idea how much i goggled it.

Offline stahta01

  • Lives here!
  • ****
  • Posts: 7670
    • My Best Post
Re: How to call win32 APIs in inlined at&t asm?
« Reply #3 on: May 01, 2022, 08:06:50 am »
« Last Edit: May 01, 2022, 08:08:35 am by stahta01 »
C Programmer working to learn more about C++ and Git.
On Windows 7 64 bit and Windows 10 64 bit.
--
When in doubt, read the CB WiKi FAQ. http://wiki.codeblocks.org

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6025
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: How to call win32 APIs in inlined at&t asm?
« Reply #4 on: May 01, 2022, 09:24:58 am »
This is the test code I use(I'm using 64bit GCC compiler)

Code
#include <iostream>

using namespace std;

char * msg = "Hello, World!\n";
char * wMsg = "Content of the window..";
char * wCaption = "Window title";



int main (){
  std::cout << "before call" << "\n";

    int src = 1;
    int dst;

    asm ("mov %1, %0\n\t"
        "add $1, %0"
        : "=r" (dst)
        : "r" (src));

    printf("%d\n", dst);


   asm(
    "movq $0, %%rax\n\t"
    "movq $0, %%rax\n\t"
    "add $48, %%rax\n\t"
    "movq $0, %%rbx\n\t"
    "movq $0, %%rcx\n\t"
    "movq $0, %%rdx\n\t"
    "push %%rax\n\t"
    "push %%rbx\n\t"
    "push %%rcx\n\t"
    "push %%rdx\n\t"
    "call MessageBoxA\n\t"
    :
    : "r" (msg), "r" (wMsg) , "r" (wCaption)
    : "cc");

    std::cout << "after call" << "\n";
}


The MessageBoxA function can be called.

But I have no idea how to pass the string to this function.

See those links as reference:

http://asm.sourceforge.net/articles/rmiyagi-inline-asm.txt

https://stackoverflow.com/questions/66323292/messagebox-program-in-x86-assembly

https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html#GotoLabels

https://www.ibiblio.org/gferg/ldp/GCC-Inline-Assembly-HOWTO.html


BTW: you need an export on assembly language for help. Our forum is not the right forum to ask.

EDIT:

To reference the input variables, you need
Code
%0,  %1 and %2
like string in the assembly code.
« Last Edit: May 01, 2022, 09:50:47 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6025
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: How to call win32 APIs in inlined at&t asm?
« Reply #5 on: May 01, 2022, 10:12:31 am »
OK, I just notice that the call convension is different in 64bit and 32bit Windows.

So, here is the modified code for 64bit Windows

Code
#include <iostream>

using namespace std;

char * msg = "Hello, World!\n";
char * wMsg = "Content of the window..";
char * wCaption = "Window title";



int main (){
  std::cout << "before call" << "\n";


   asm(
    "movq $0, %%rcx\n\t"
    "movq %2, %%rdx\n\t"
    "movq %1, %%r8\n\t"
    "movq $0, %%r9\n\t"
    "callq MessageBoxA\n\t"
    :
    : "r" (msg), "r" (wMsg) , "r" (wCaption)
    : "cc");

    std::cout << "after call" << "\n";
}

The important change is: for 64bit function call, the first 4 arguments are not pushed in the stack, but put in the registers, see some references:

https://en.wikipedia.org/wiki/X86_calling_conventions#Microsoft_x64_calling_convention

https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170

https://stackoverflow.com/questions/42488273/call-a-function-with-inline-asm
« Last Edit: May 01, 2022, 10:14:15 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline reverser

  • Multiple posting newcomer
  • *
  • Posts: 17
Re: How to call win32 APIs in inlined at&t asm?
« Reply #6 on: May 07, 2022, 07:48:53 am »
OK, I just notice that the call convension is different in 64bit and 32bit Windows.

thanks a lot. it executed the call but Args are not passed correctly as message box is titled "Error" and is empty.
can you help me with 32bit version too; in x32, it shows the same error i mentioned. i cant make it work. if you cant (as you said here is not the right forum), please PM me the right place of asking this.

regards.
« Last Edit: May 07, 2022, 09:13:52 am by reverser »

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6025
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: How to call win32 APIs in inlined at&t asm?
« Reply #7 on: May 07, 2022, 03:15:55 pm »
OK, I just notice that the call convension is different in 64bit and 32bit Windows.

thanks a lot. it executed the call but Args are not passed correctly as message box is titled "Error" and is empty.
can you help me with 32bit version too; in x32, it shows the same error i mentioned. i cant make it work. if you cant (as you said here is not the right forum), please PM me the right place of asking this.

regards.

I'm not sure, you can just Google for some asm forum(maybe masm32.com or others). Maybe stackoverflow site is also OK.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline reverser

  • Multiple posting newcomer
  • *
  • Posts: 17
Re: How to call win32 APIs in inlined at&t asm?
« Reply #8 on: June 25, 2022, 08:16:32 am »
OK, I just notice that the call convension is different in 64bit and 32bit Windows.

thanks a lot. it executed the call but Args are not passed correctly as message box is titled "Error" and is empty.
can you help me with 32bit version too; in x32, it shows the same error i mentioned. i cant make it work. if you cant (as you said here is not the right forum), please PM me the right place of asking this.

regards.
to resolve this problem:
in x86 mode, win32 APIs must be called by their decorated names, preceded by a under score e.g calll _messageBox@16 (16 is the number of bytes that the functions parameters use on stack)

Offline benkenobi01

  • Multiple posting newcomer
  • *
  • Posts: 64
Re: How to call win32 APIs in inlined at&t asm?
« Reply #9 on: July 12, 2022, 07:47:59 pm »
You're wasting your time with that, install masm or nasm and do real asm.