Author Topic: stop at ntdll!DbgUiConnectToDbg () on debug start  (Read 24278 times)

Offline Hitnrun

  • Multiple posting newcomer
  • *
  • Posts: 10
stop at ntdll!DbgUiConnectToDbg () on debug start
« on: May 15, 2008, 04:39:18 pm »
I have a project, that everytime I start in debug mode, I get:

Code
Program received signal SIGTRAP, Trace/breakpoint trap.
In ntdll!DbgUiConnectToDbg () (ntdll.dll)

If I press Ctrl-F7 about 50 (!) times, it runs, but at various points on the application I get it again.

There are no breapoints set on CodeBlocks, why is it stopping at a breakpoint trap?

Offline Hitnrun

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #1 on: May 18, 2008, 04:32:14 pm »
Running in console mode gdb, I get a more detailed message:


Code
Invalid Address specified to RtlFreeHeap

All the times, looks like it is when including an external DLL (FreeImage).

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #2 on: May 19, 2008, 09:12:34 am »
Though it's impossible to tell the real reasons, I see two possibilities.
1. variables not being initialised properly are the most common reason for "only crashes in debugger" kind of error.
2. "Invalid Address specified to RtlFreeHeap" is sometimes encountered in otherwise correct code, if there is an executable and a DLL, one of which was built in debug mode and the other one in release mode. That will create two different heaps managed by two heap managers, which will cause heap corruption whenever pointers are handed from one to the other (so basically, all the time in most programs). This may generate an error which the debugger catches, or may go silently otherwise, or may crash at a random later time.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Hitnrun

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #3 on: May 19, 2008, 04:51:26 pm »
2. "Invalid Address specified to RtlFreeHeap" is sometimes encountered in otherwise correct code, if there is an executable and a DLL, one of which was built in debug mode and the other one in release mode. That will create two different heaps managed by two heap managers, which will cause heap corruption whenever pointers are handed from one to the other (so basically, all the time in most programs). This may generate an error which the debugger catches, or may go silently otherwise, or may crash at a random later time.

Hmm the DLL was built by VC2005, but the program is using mingw, does this apply in this case too? Would building the DLL as debug in VC overcome heap corruption problems in mingw applications?

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #4 on: May 19, 2008, 07:51:41 pm »
Good question, I don't know what MinGW uses for memory allocation/deallocation.
But building the DLL in debug mode would be my first try, since debug+release is known to cause problems in any case.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Rytz

  • Single posting newcomer
  • *
  • Posts: 9
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #5 on: July 28, 2008, 11:15:28 am »
I had an issue with the boost filesystem library that raised the same error only when I was debugging it.  I was able to fix the issue by setting the program to run on one processor (I'm running a dual-core system).  You can do that permanently (at least in XP) by setting the program compatibility mode on the executable or you can set the affinity at run-time by using the task manager (I think I still got the error with this method though).

I saw this thread before when I was having my problem so I figured I'd share this little tidbit :).

Offline nanyu

  • Almost regular
  • **
  • Posts: 188
  • nanyu
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #6 on: April 14, 2009, 01:59:15 pm »
I found the answer from http://n2.nabble.com/Invalid-Address-specified-to-RtlFreeHeap-and-libstdc%2B%2B-problem-td2092059.html

Code::Blocks does allow you to automatically send commands to gdb after it
starts though.  Select "Compiler and debugger ..." from the Settings
menu.  Click on "Debugger settings" in the left pane of the window that
pops up.  Enter the commands you want to send to gdb in the "Debugger
initialization commands" textfield in the right pane.  (And entering
"handle SIGTRAP noprint" worked for me.)

Fett

  • Guest
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #7 on: June 28, 2010, 08:06:10 am »
I had the same problem. I solved it, when export my project in m$ vs, and vs's debugger show me where i did mistake- it was then i use "delete" statment. i hope in future mingw32, and gdb will have a more(maybe most) powerful dev tools   :).

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5914
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #8 on: June 28, 2010, 09:21:24 am »
I had the same problem. I solved it, when export my project in m$ vs, and vs's debugger show me where i did mistake- it was then i use "delete" statment. i hope in future mingw32, and gdb will have a more(maybe most) powerful dev tools   :).

Which gdb version did you use? I guess you have an old version of MinGW GDB, so please update it first. And if you post your test code, I'm like to test for you. :D ( I don't have this problem)
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline eigen

  • Single posting newcomer
  • *
  • Posts: 4
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #9 on: December 15, 2010, 11:24:44 am »
Sorry to bring this old topic up, but I have a similar issue. When starting an app in debugger, or any app in debug mode, first I get TWO (always) error messages in a row:



and then the debugger tells me

#0 7C964ED1   ntdll!RtlReAllocateHeap() (C:\WINDOWS\system32\ntdll.dll:??)
#1 7C93F8FB   ntdll!RtlInsertElementGenericTableAvl() (C:\WINDOWS\system32\ntdll.dll:??)
#2 7C90EAC7   ntdll!LdrCreateOutOfProcessImage() (C:\WINDOWS\system32\ntdll.dll:??)

If I put gdb.exe in the app folder and drag the exe onto it, I get the RtlReAllocateHeap error. But when I do the same in my gf's computer, all works as expected. So, something must be totally messed up in mine.

The thing is, a simple console "Hello world" works fine in the debugger, but as soon as the app has to get ANYTHING from an external dll, it crashes in debug mode. Even an additional line of SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER); is enough to throw the RtlReAllocateHeap error when starting the debugger.

Everything is working in release mode, as well in the debug mode a few days ago. I don't think I have changed anything in the system lately .. So, have you guys got any ideas what it could be? As a last resort, I'm thinking of re-installing Windows, in case there's something messed up internally. Maybe the ntdll.dll is somehow broken, though the file size is exactly as it should be, so I don't think it's been modified.

Thanks in advance,
Eigen
« Last Edit: December 15, 2010, 11:57:33 am by eigen »

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5914
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #10 on: December 15, 2010, 01:12:37 pm »
Sorry to bring this old topic up, but I have a similar issue. When starting an app in debugger, or any app in debug mode, first I get TWO (always) error messages in a row:



and then the debugger tells me

#0 7C964ED1   ntdll!RtlReAllocateHeap() (C:\WINDOWS\system32\ntdll.dll:??)
#1 7C93F8FB   ntdll!RtlInsertElementGenericTableAvl() (C:\WINDOWS\system32\ntdll.dll:??)
#2 7C90EAC7   ntdll!LdrCreateOutOfProcessImage() (C:\WINDOWS\system32\ntdll.dll:??)

If I put gdb.exe in the app folder and drag the exe onto it, I get the RtlReAllocateHeap error. But when I do the same in my gf's computer, all works as expected. So, something must be totally messed up in mine.

The thing is, a simple console "Hello world" works fine in the debugger, but as soon as the app has to get ANYTHING from an external dll, it crashes in debug mode. Even an additional line of SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER); is enough to throw the RtlReAllocateHeap error when starting the debugger.

Everything is working in release mode, as well in the debug mode a few days ago. I don't think I have changed anything in the system lately .. So, have you guys got any ideas what it could be? As a last resort, I'm thinking of re-installing Windows, in case there's something messed up internally. Maybe the ntdll.dll is somehow broken, though the file size is exactly as it should be, so I don't think it's been modified.

Thanks in advance,
Eigen
Hi, I have no idea about your problem, but at least you should tell us
OS?
CB?
debugger?
GCC?
all these information and the version???

and what is the test code?
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline eigen

  • Single posting newcomer
  • *
  • Posts: 4
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #11 on: December 15, 2010, 02:27:00 pm »
Windows XP SP2
Code::Blocks 10.05
GNU gdb 7.2 (same thing with the default debugger that comes with CB 10.05 .. so that's not the culprit)
MinGW (whatever the version CB comes with)

Well, test code .. even a simple app like this crashes.

Code
#include <windows.h>
#include "SDL.h"

int WINAPI WinMain(HINSTANCE d1, HINSTANCE d2, LPSTR d3, int d4)
{
    SDL_Init(SDL_INIT_VIDEO);

    printf("Hello World!");

    SDL_Quit();

    return 0;
}

.. where as this ..

Code
#include <windows.h>
#include "SDL.h"

int WINAPI WinMain(HINSTANCE d1, HINSTANCE d2, LPSTR d3, int d4)
{
    //SDL_Init(SDL_INIT_VIDEO);

    printf("Hello World!");

    //SDL_Quit();

    return 0;
}

.. doesn't. I get "Program exited normally. Debugger finished with status 0" as expected.

It's not only with SDL .. I also use Irrlicht graphics engine which produces the same error as soon as I call any function in an external DLL file. They may not be debug builds, but everything was working 2 days ago and it definitely shouldn't cause errors like that.

Okay .. I just realized that the SDL.dll is not even used, so it's a static library. Okay, what the hell is going on ..

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5914
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #12 on: December 15, 2010, 02:47:01 pm »
I do not have sdl library ( I even don't understand what SDL libraries used for  :D)

but it seems the  called function has some problems:

Code
SDL_Init(SDL_INIT_VIDEO);

does the program works ok without gdb??
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline eigen

  • Single posting newcomer
  • *
  • Posts: 4
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #13 on: December 15, 2010, 03:04:08 pm »
That function should not have any problems. It was just a quick sample without even creating a real window, but yes, every program works on my and everyone elses computer. Also, the debugging works everywhere else but mine. I guess I have to reinstall Windows when I get the time and see if it helps.

I tried WinDbg, a tool I got from Microsoft site and attached it to my running project process and immediately it threw erros related to ntdll and also said symbols for that file could not be found. I'll try downloading and reinstalling the symbol files package for XP SP2 from here http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx#f .. maybe it helps.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5914
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #14 on: December 15, 2010, 03:16:37 pm »
That function should not have any problems. It was just a quick sample without even creating a real window, but yes, every program works on my and everyone elses computer. Also, the debugging works everywhere else but mine. I guess I have to reinstall Windows when I get the time and see if it helps.

I tried WinDbg, a tool I got from Microsoft site and attached it to my running project process and immediately it threw erros related to ntdll and also said symbols for that file could not be found. I'll try downloading and reinstalling the symbol files package for XP SP2 from here http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx#f .. maybe it helps.
that weird. I don't think the debug symbols files were necessary. :D
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Pecan

  • Plugin developer
  • Lives here!
  • ****
  • Posts: 2775
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #15 on: December 15, 2010, 03:35:15 pm »
Googling "0xc0000235" produces a bunch of messages that imply that there is a conflict between accesses to some dll's and anti-virus updates.

Google "0xc0000235". You might find a situation matching yours.

Offline eigen

  • Single posting newcomer
  • *
  • Posts: 4
Re: stop at ntdll!DbgUiConnectToDbg () on debug start
« Reply #16 on: December 15, 2010, 03:46:42 pm »
Thanks. I guess Avira AntiVir (which I also have installed) has messed something up with some of their updates. I hope this gets fixed soon.

edit: got a solution from https://forum.avira.com/wbb/index.php?page=Thread&postID=1031807 .. thanks for the tip, Pecan!!
« Last Edit: December 15, 2010, 04:04:44 pm by eigen »