[SOLVED] How to debug dll

[DanRR]

Hello,
I didn't found out how to set the debugger to debug a dll. This is my first windows dll.
I'm using 5456 build, I've set a dll project, using the default settings and a application project, to use it.
I linked the dll statically, by adding *.a dll file. The application works well with the dll when executed directly.
Pressing the Debug button on the dll issue the obvious message that I need a host program to run it, how do I do that?

Thanks

[Pecan]

MainMenu->Project->Set Programs Aruguments->Host Applicaton

Be aware that setting breakpoints before loading the DLL may not be honored  by gdb and that you will have to insert an "asm("int3"); /*trap*/" statement into the code to get it to break in the debugger first, then set your breakpoints.

[DanRR]

Thanks Pecan!
That helped, The debugger is breaking in the dll now.
There are still some problems:
1. I can't break debug some initiation code called on DLL_PROCESS_ATTACH event, I think it's because the
trap stops the debugger after the dll is loaded.
2. I can't break on DLL_PROCESS_DETACH, I'm not sure why.
Is there a solution to these problems?
 

[Pecan]

Thanks Pecan!
That helped, The debugger is breaking in the dll now.
There are still some problems:
1. I can't break debug some initiation code called on DLL_PROCESS_ATTACH event, I think it's because the
trap stops the debugger after the dll is loaded.
2. I can't break on DLL_PROCESS_DETACH, I'm not sure why.
Is there a solution to these problems?
 

You do hit continue or NextLine after the debugger stops at the asm("int3") statement don't you? I've had no problem with this.

If a breakpoint doesn't work in a DLL, I simple always resort to the asm("int3") trick. I cannot tell you why breakpoints do or don't work at particular places.

If you mean that you already have an asm("int3)" in DLL_PROCESS_DETACH and it doesn't break into gdb, I'd investidate  why that code is never being executed.
 

[DanRR]

Thanks Pecan!
That helped, The debugger is breaking in the dll now.
There are still some problems:
1. I can't break debug some initiation code called on DLL_PROCESS_ATTACH event, I think it's because the
trap stops the debugger after the dll is loaded.
2. I can't break on DLL_PROCESS_DETACH, I'm not sure why.
Is there a solution to these problems?
 

You do hit continue or NextLine after the debugger stops at the asm("int3") statement don't you? I've had no problem with this.

If a breakpoint doesn't work in a DLL, I simple always resort to the asm("int3") trick. I cannot tell you why breakpoints do or don't work at particular places.

If you mean that you already have an asm("int3)" in DLL_PROCESS_DETACH and it doesn't break into gdb, I'd investidate  why that code is never being executed.
 

I managed to brake in the dll functions, using asm("int3") in the host program.
I couldn't break in any of the dll events, using breakpoint or asm("int3)". I gave up on this one, I'll workaround that.
Thanks again!

[ollydbg]

You don't need to use "int 3" like asm code, see the attachment below, you can directly set breakpoint in the DLL source.


[attachment deleted by admin]

[DanRR]

You don't need to use "int 3" like asm code, see the attachment below, you can directly set breakpoint in the DLL source.

Hi ollydbg,
I tested your example project, indeed, asm("int3") is not needed when using dynamic dll linking.
Of course, it takes some extra work to link the dll dynamically.
Thanks!