Author Topic: Batch build problems (hangs/stalls if log window is not active)  (Read 17446 times)

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Batch build problems (hangs/stalls if log window is not active)
« on: September 18, 2018, 05:33:03 pm »
Hi all,

I am trying to use the batch build feature of Codeblocks to have my projects built automatically on a Jenkins CI server. This may be related to some earlier discussions:
- http://forums.codeblocks.org/index.php/topic,21838.msg148541.html#msg148541 (Linux and assumingly outdated solution "special build by Jens")
- http://forums.codeblocks.org/index.php/topic,14928.0.html (Linux solution: export DISPLAY=:2000.0)
- http://forums.codeblocks.org/index.php?topic=15662.0 (codeblocks_console branch, found it on github, assumingly outdated, could not get it to compile)

The problem that I get is on MS Windows 7 and with projects that produce *a lot* of compiler output (template-heavy code). Here is what happens:
- Running codeblocks with a command like this: codeblocks.exe <PATH_TO_WORKSPACE> --rebuild --no-check-associations --no-dde --no-splash-screen --no-batch-window-close --target=<TARGET>
- The batch build window opens and the build starts (I can see compiler output in the log window and cl.exe or mingw-gcc.exe shows up in the task manager's process list consuming cpu time and memory)
- After a while of not activating the batch log window, the build stalls, there is no more compiler output being appended and the compiler executables cpu usage goes down to zero
- Only if I hover over or click on the batch log window, the build continues, I can see new compiler output and cpu time being consumed
- Leaving the batch log window minimized or just not focused, the build stalls again

I already tried a couple of things:
- I looked into the Codeblocks code trying to find an easy way to "deactivate" the batch log window.. my current conclusion is that there is none ;)
- I managed to implement a hack that disables printing all the compiler output into the log window (remember the problem seems to only occur with projects that produce *much* output). The window is empty now, but the problem persists
- I tried to get a backtrace using gdb, which I don't know very well so I'm not sure if I did it right. After starting up gdb with codeblocks.exe as the debugee and waiting for the build to stall, I had to use a little helper application (http://www.mingw.org/wiki/Workaround_for_GDB_Ctrl_C_Interrupt) to interrupt the process and to be able to get a backtrace

Here is the result of "(gdb) thread apply all bt":
Code
(gdb) thread apply all bt

Thread 146 (Thread 6524.0x1b84):
#0  0x77d5000d in ntdll!DbgBreakPoint () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77ddf306 in ntdll!DbgUiRemoteBreakin () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x3e10b887 in ?? ()
#3  0x00000000 in ?? ()

Thread 145 (Thread 6524.0x1ac4):
#0  0x77d61f76 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d61f76 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x77d8fa8e in ntdll!TpSetTimer () from C:\WINDOWS\SysWOW64\ntdll.dll
#3  0x7780343d in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\syswow64\kernel32.dll
#4  0x77d79832 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#5  0x77d79805 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#6  0x00000000 in ?? ()

Thread 143 (Thread 6524.0x1cec):
#0  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x76ed15ce in WaitForSingleObjectEx () from C:\WINDOWS\syswow64\KernelBase.dll
#3  0x000001b0 in ?? ()
#4  0x00000000 in ?? ()

Thread 51 (Thread 6524.0x22b0):
---Type <return> to continue, or q <return> to quit---
#0  0x77d61f76 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d61f76 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x77d8fa8e in ntdll!TpSetTimer () from C:\WINDOWS\SysWOW64\ntdll.dll
#3  0x7780343d in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\syswow64\kernel32.dll
#4  0x77d79832 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#5  0x77d79805 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#6  0x00000000 in ?? ()

Thread 47 (Thread 6524.0x9c4):
#0  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x76ed15ce in WaitForSingleObjectEx () from C:\WINDOWS\syswow64\KernelBase.dll
#3  0x00000290 in ?? ()
#4  0x00000000 in ?? ()

Thread 46 (Thread 6524.0x17f0):
#0  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x76ed15ce in WaitForSingleObjectEx () from C:\WINDOWS\syswow64\KernelBase.dll
#3  0x0000032c in ?? ()
#4  0x00000000 in ?? ()

Thread 45 (Thread 6524.0x86c):
#0  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
---Type <return> to continue, or q <return> to quit---
#1  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x76ed15ce in WaitForSingleObjectEx () from C:\WINDOWS\syswow64\KernelBase.dll
#3  0x000001e8 in ?? ()
#4  0x00000000 in ?? ()

Thread 2 (Thread 6524.0x1670):
#0  0x77d6018d in ntdll!ZwWaitForMultipleObjects () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d6018d in ntdll!ZwWaitForMultipleObjects () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x77d8f69f in ntdll!_allmul () from C:\WINDOWS\SysWOW64\ntdll.dll
#3  0x00000003 in ?? ()
#4  0x014ecf50 in ?? ()
#5  0x7780343d in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\syswow64\kernel32.dll
#6  0x77d79832 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#7  0x77d79805 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#8  0x00000000 in ?? ()

Thread 1 (Thread 6524.0x10f8):
#0  0x767578d7 in USER32!DispatchMessageW () from C:\WINDOWS\syswow64\user32.dll
#1  0x627df5cc in wxEventLoop::Dispatch() ()
   from D:\Users\scho_so\Dev\codeblocks-17.12-BatchFix\src\devel\wxmsw28u_gcc_custom.dll
#2  0x628b1798 in wxEventLoopManual::Run() ()
   from D:\Users\scho_so\Dev\codeblocks-17.12-BatchFix\src\devel\wxmsw28u_gcc_custom.dll
#3  0x0a9a9448 in ?? ()
#4  0x8775d038 in ?? ()
---Type <return> to continue, or q <return> to quit---
#5  0x3128c483 in ?? ()
#6  0x04c25bc0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Does anyone have ideas about ...
... where the problem with a stalling build log window could originate from (e.g. wxWidgets modal windows / event loops etc.) ?
... how to further identify the problem, e.g. using some more advanced debugging techniques?
... things to try in order to find a work-around (e.g. some 'hackish' solution until someone can get a grip of the actual problem) ?

Help is much appreciated here and I would be more than willing to try out possible solutions or debugging techniques. Unfortunately, I haven't been able to reproduce the problem in a sort of "minimal example" that I could share, since it only comes up with certain projects that produce a lot of template-tracing compiler outputs. That's also the reason, I haven't issued a bug report...

Thanks everyone.


Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #2 on: September 18, 2018, 06:39:07 pm »
simons: Can you try to disable the option in the cmd.exe's settings? If it works we'll add it to the code, so it is automatic.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #3 on: September 19, 2018, 09:17:59 am »
Thanks for the quick response.

I did not have the "quick edit mode" enabled in my command prompt. Anyway, I tried to enable/disable it without any changes to the symptoms.

Just to clarify: It's the Codeblocks batch build log window (a little wxWidgets modal dialog, as far as I read from the sources) that stalls and doesn't show any more compiler output. It's *not* the command prompt. Actually Codeblocks - even in batch mode - seems to completely detach from the command prompt such that I can close the command prompt window and just have the Codeblocks batch build log window open. It is this window that seems to stall the compile process when not focused for a while. Any GUI event, even hovering over the window with the mouse or dragging another window over it, will continue the build process. As soon as I put my hands off the mouse and the window does not have focus, it stalls again.

Any more ideas?

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #4 on: September 19, 2018, 12:06:07 pm »
It could be the idle handler we're probably using.
Can you reproduce it with an open source project?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Miguel Gimenez

  • Developer
  • Lives here!
  • *****
  • Posts: 1553
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #5 on: September 19, 2018, 02:26:00 pm »
Checking this issue I found another problem. After the reuild is finished, I close the window and a SIGSEGV is received:

Code
Thread 1 received signal SIGSEGV, Segmentation fault.
0x0046db13 in InfoPane::Show (this=0x8861a68, logger=0xbd81cd8) at G:\Codeblocks\src\src\infopane.cpp:253
253             if (m_Pages.Item(i)->logger == logger)
(gdb) bt
#0  0x0046db13 in InfoPane::Show (this=0x8861a68, logger=0xbd81cd8) at G:\Codeblocks\src\src\infopane.cpp:253
#1  0x00496b96 in MainFrame::OnSwitchToLogWindow (this=0x87a6670, event=...) at G:\Codeblocks\src\src\main.cpp:5036
#2  0x00513a3c in cbEventFunctor<MainFrame, CodeBlocksLogEvent>::Call (this=0x8845038, event=...)
    at G:/Codeblocks/src/include/cbfunctor.h:49
#3  0x01fe7833 in Manager::ProcessEvent (this=0x7ea7310, event=...) at G:\Codeblocks\src\sdk\manager.cpp:328
#4  0x6a157d8b in CompilerGCC::ClearLog (this=0x88f3878) at G:\Codeblocks\src\plugins\compilergcc\compilergcc.cpp:1061
#5  0x6a16ab0b in CompilerGCC::OnWorkspaceClosed (this=0x88f3878, event=...)
    at G:\Codeblocks\src\plugins\compilergcc\compilergcc.cpp:3405
#6  0x6a1d5b3c in cbEventFunctor<CompilerGCC, CodeBlocksEvent>::Call (this=0xa10f358, event=...)
    at G:/Codeblocks/src/include/cbfunctor.h:49
#7  0x01fe7581 in Manager::ProcessEvent (this=0x7ea7310, event=...) at G:\Codeblocks\src\sdk\manager.cpp:260
#8  0x02002507 in PluginManager::NotifyPlugins (this=0x88e16e0, event=...) at G:\Codeblocks\src\sdk\pluginmanager.cpp:1522
#9  0x020297ac in ProjectManager::CloseWorkspace (this=0x885ede8) at G:\Codeblocks\src\sdk\projectmanager.cpp:613
#10 0x0047fd78 in MainFrame::DoCloseCurrentWorkspace (this=0x87a6670) at G:\Codeblocks\src\src\main.cpp:1877
#11 0x0048945b in MainFrame::OnApplicationClose (this=0x87a6670, event=...) at G:\Codeblocks\src\src\main.cpp:2758
#12 0x644433d2 in wxAppConsoleBase::HandleEvent(wxEvtHandler*, void (wxEvtHandler::*)(wxEvent&), wxEvent&) const ()
   from C:\Windows\system32\wxmsw311u_gcc_custom.dll
#13 0x0028f5f0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) print i
$1 = 0
(gdb) print m_Pages
$2 = {<wxBaseArrayPtrVoid> = {m_nSize = 4277075694, m_nCount = 4277075694, m_pItems = 0xfeeefeee}, <No data fields>}

I think m_Pages has not been initialized at this point. 4277075694 = 0xfeeefeee.

EDIT: the attached patch fixes this, I'll create a ticket.
« Last Edit: September 19, 2018, 02:46:05 pm by Miguel Gimenez »

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #6 on: September 19, 2018, 05:09:42 pm »
I managed to assemble a "somewhat minimal" example using boost::msm (template-heavy code) that reproduces the problem when compiling with visual studio 2005 (msvc8) or visual studio 2010 (msvc10).

It seems like there are a couple conditions that raise the probability of the problem: many compiler warnings and errors, multiple compiler processes, a decent cpu load on the compiler. I also played around with some code snippets from https://codegolf.stackexchange.com/questions/5802/write-the-shortest-program-that-generates-the-most-compiler-warnings-and-errors and while I managed to totally hog the Codeblocks application with some of these "compiler bombs", I could not reproduce my original problem. Also, bare in mind that I have seen the issue with only a single compiler process as well.

I uploaded the "not-so-minimal" example here: https://github.com/simfinite/test_codeblocks_batch_build
Please, follow the steps in the README to reproduce the problem (and sorry for making you download the boost sources...  ;))

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #7 on: September 19, 2018, 05:45:07 pm »
Can you reproduce this with mingw? The older versions of Visual studio aren't made to run in parallel.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #8 on: September 19, 2018, 10:47:11 pm »
I can reproduce this on my machine with windows7 mingw-tdm-5 and the example project... Funny...
For me it always stops at linking stage. I am the only one?

I can not reproduce the SIGSEGV from Miguel Gimenez

There really should be a codeblocks console builder... It is nearly a shame that this does not exists..

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #9 on: September 20, 2018, 10:36:32 am »
Quote
I can reproduce this on my machine with windows7 mingw-tdm-5

Interesting, I could not reproduce it with gcc on my machine yesterday. But I added a little warning generator to the example code and now I can reproduce the problem with
- msvc8
- msvc10
- TDM gcc 5.1.0
- MinGW-W64 gcc 5.4.0
(all with 4 build jobs)

The problem is still most easily reproduced with the old and weak visual studio compilers and hardest with gcc 5.4.0. It may take some tweaking of the amount of warnings generated (see fsm.hpp, line #206) or the number of build jobs. Although this sounds like an undeterministic problem, I could reproduce it *every time* once I found a configuration that works.


Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #10 on: September 20, 2018, 12:17:46 pm »
i would like to add, that i have enabled -wall and -wpedantic to generate even more messages

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #11 on: September 20, 2018, 06:02:12 pm »
Can you write a simple app that just spits output which looks like warnings generated by GCC and set this as the compiler executable?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #12 on: September 21, 2018, 12:16:02 am »
Can you write a simple app that just spits output which looks like warnings generated by GCC and set this as the compiler executable?
tried it right now with > 10000 warnings and i am not able to trigger this. The output count should be a lot more then from the working project. At least it needs a lot more time to ouput.. There has to be something else contributing to this...

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #13 on: September 21, 2018, 09:54:07 am »
Quote
tried it right now with > 10000 warnings and i am not able to trigger this
Yes, my observation is that the problem occurs only for the combination of many warnings being produced *and* heavy CPU load from the compiler processes. That's the reason I included boost::msm into the example application, because the simple warning generator wasn't enough and compiling the template-heavy library code consumes a lot of CPU time. The sample application pretty much reproduces the setting that I have in the projects where I originally observed the problem.

Quote
Can you write a simple app that just spits output which looks like warnings generated by GCC and set this as the compiler executable?
Yes, I wrote a little "fake_gcc" and put it into the same repository. I was able to reproduce the problem with it, however, some tweaking may still be required. There are some variables inside the main.cpp that configure how the program behaves. They are set to what worked on my machine. If you are having trouble reproducing the problem, try to increase/decrease "work_us" to increase/decrease CPU time consumption and check that the processes are in fact consuming considerable CPU time (e.g. 15-25% each with 4 build jobs on a quad core).

One thing I noticed using the "fake_gcc": The problem seems to only arise when compiler processes start up and/or quit. For example, I could not reproduce the problem just letting fake_gcc run forever. The project being built also needs to have many compilation units, such that new compiler processes are started up every now and then.
« Last Edit: September 21, 2018, 12:59:49 pm by simons »

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #14 on: September 27, 2018, 09:28:11 am »
I did some more experiments and have some minor insights to share:

- As I mentioned before, up until now I could only reproduce the problem if the compiler process is consuming considerable CPU time. I verified that the problem cannot be reproduced by hogging the CPU with some other processes and letting the compiler process output warnings only.

- However, now I found a configuration for the "fake_gcc"  that reproduces the problem without the busy loop, i.e. without consuming CPU time and by but just letting the compiler process sleep for a while each iteration. It's a bit harder to reproduce than the variant with the busy loop and only works with 4 build jobs on my machine. Still this decouples the problem from CPU usage and is pointing more towards a general timing issue.

Has anyone been able to reproduce the problem using the "fake_gcc"? I pushed the new configuration to the repo. Any ideas on how to move on from here?

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #15 on: September 27, 2018, 03:53:49 pm »
Has anyone been able to reproduce the problem using the "fake_gcc"? I pushed the new configuration to the repo. Any ideas on how to move on from here?
no, i was not able to reproduce it with your latest fake_gcc...

I have tried future with WinDBG and gcc and i got this information:
cc1plus.exe:
Code
DEFAULT_BUCKET_ID:  APPLICATION_HANG_BlockedOn_FileIO

PRIMARY_PROBLEM_CLASS:  BlockedOn_FileIO

THREAD_SHA1_HASH_MOD_FUNC:  c281b8792a9b0b7b09f56f4b37bca845ce1f1b92

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  5428a93740841021c6fded59898de991617bd52a

LAST_CONTROL_TRANSFER:  from 7618e004 to 773af96d

STACK_TEXT: 
0028f340 7618e004 00000044 00000000 00000000 ntdll!NtWriteFile+0x15
0028f3a4 76ca12cc 00000044 0028f418 00000400 KERNELBASE!WriteFile+0x113
0028f3c0 76dd4236 00000044 0028f418 00000400 kernel32!WriteFileImplementation+0x76
0028f97c 76dd40eb 00000002 12b99598 00000877 msvcrt!_write_nolock+0x3fb
0028f9c0 76dcf15e 00000002 12b99598 00000877 msvcrt!_write+0x9f
0028f9e0 76dcf1a0 76e62940 00000000 0028fa28 msvcrt!_flush+0x3b
0028f9f0 76dd416d 76e62940 ff7a9eca 01024a9c msvcrt!_fflush_nolock+0x1c
0028fa28 00ea85fd 76e62940 0028fb18 0028fb18 msvcrt!fflush+0x30
WARNING: Stack unwind information not available. Following frames may be wrong.
0028faf8 00ea505e 01463500 0028fb18 00000034 cc1plus+0xaa85fd
0028fb48 00429b48 000001aa 01024a9c 15d3a2c0 cc1plus+0xaa505e
0028fb58 0043f039 15153910 00000000 01bc5540 cc1plus+0x29b48
0028fbb8 004cc7c0 00000001 00000001 00000000 cc1plus+0x3f039
0028fbd8 004cd4f1 1917b708 00465250 0c95bf80 cc1plus+0xcc7c0
0028fbf8 0043fd61 1917b708 00000000 016b9cc0 cc1plus+0xcd4f1
00000000 00000000 00000000 00000000 00000000 cc1plus+0x3fd61

mingw32g++.exe:
Code
DERIVED_WAIT_CHAIN:    00 ca8.24c8 Process Handle 

WAIT_CHAIN_COMMAND:  ~0s;k;;

THREAD_ATTRIBUTES:
BLOCKING_THREAD:  000024c8

DEFAULT_BUCKET_ID:  APPLICATION_HANG_BlockedOn_ProcessHandle

PRIMARY_PROBLEM_CLASS:  BlockedOn_ProcessHandle

THREAD_SHA1_HASH_MOD_FUNC:  e48a380cac1320f60a2487066e7609815690d73a

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  bcd69aa1f032d972611d5e1bfe1a686de0b4cd74

LAST_CONTROL_TRANSFER:  from 761915ce to 773af901

STACK_TEXT: 
0028f490 761915ce 00000054 00000000 00000000 ntdll!NtWaitForSingleObject+0x15
0028f4fc 76ca1194 00000054 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
0028f514 76ca1148 00000054 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
0028f528 0044c64b 00000054 ffffffff 0028f6a8 kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0028f538 00444aa8 00000004 00360100 0036fe33 mingw32_g__+0x4c64b
0028f6a8 0040a4c5 0036c368 004f4220 0000002f mingw32_g__+0x44aa8
0028f738 0040ca8e 003688f9 004f4aa3 00000001 mingw32_g__+0xa4c5
0028f79c 773c3476 77981bc8 0047fe4a 004f4aa1 mingw32_g__+0xca8e
0028f938 00401577 004f3470 0047ff78 00000019 ntdll!RtlpAllocateHeap+0xe66
0028f958 0040a917 0047c9b4 00361c32 00000009 mingw32_g__+0x1577
0028f9e8 0040ca8e 003688f9 004f3fdb 0000000c mingw32_g__+0xa917
0028fa48 00406a05 773c3476 77981898 004f3bcd mingw32_g__+0xca8e
0028faf8 0040ca8e 003688f9 004f3d3b 00000002 mingw32_g__+0x6a05
0028fb5c 773c3476 77981f88 0047db8d 004f3d39 mingw32_g__+0xca8e
0028fc88 0040b295 00000002 00000002 773c4355 ntdll!RtlpAllocateHeap+0xe66
0028fd18 0040ca8e 003688f9 0047db8b 00000001 mingw32_g__+0xb295
0028fd74 773c3476 003c34a1 00000001 00000010 mingw32_g__+0xca8e
0028fe48 0040e0b0 00000000 00000000 004f1a58 ntdll!RtlpAllocateHeap+0xe66
0028fe78 0040e1d8 0047db48 00000001 0028ffc4 mingw32_g__+0xe0b0
0028ff18 004010fd 0028ff28 76dd9e34 fffde000 mingw32_g__+0xe1d8
0028ff94 773c9832 fffde000 77981c68 00000000 mingw32_g__+0x10fd
0028ffd4 773c9805 00401280 fffde000 00000000 ntdll!__RtlUserThreadStart+0x70
0028ffec 00000000 00401280 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b

codeblocks.exe:
Code
DERIVED_WAIT_CHAIN:    00 26b8.1fbc Unknown 

WAIT_CHAIN_COMMAND:  ~0s;k;;

THREAD_ATTRIBUTES:
BLOCKING_THREAD:  00001fbc

THREAD_SHA1_HASH_MOD_FUNC:  71431138aca2c5693a1cab1a8c383575c7f4d911

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  ca61d71375c8d331e524a6f7a93b8a1e35ccd897

LAST_CONTROL_TRANSFER:  from 76ea790d to 76ea78d7

STACK_TEXT: 
0028eefc 76ea790d 0028f0b8 00000000 00000000 user32!NtUserGetMessage+0x15
0028ef18 0ca4dfd6 0028f0b8 00000000 00000000 user32!GetMessageW+0x33
WARNING: Stack unwind information not available. Following frames may be wrong.
0028ef38 0c24f8a3 0028f0b8 00000000 00000000 wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+0x26
0028f048 0c35a7aa 0028f0b8 0028f138 0028f068 wxmsw30ud_gcc_custom!ZN18wxMSWEventLoopBase14GetNextMessageEP6tagMSG+0x77
0028f0f8 0c1947e9 00000000 0108fb44 0000032c wxmsw30ud_gcc_custom!ZN14wxGUIEventLoop8DispatchEv+0x52
0028f118 0c1948b1 0028f138 00001fbc 00000000 wxmsw30ud_gcc_custom!ZN17wxEventLoopManual13ProcessEventsEv+0x4d
0028f198 0c194503 0028f210 19e5925c 00000000 wxmsw30ud_gcc_custom!ZN17wxEventLoopManual5DoRunEv+0xc5
0028f238 0cac3ad3 00000024 0028f2cc 0028f258 wxmsw30ud_gcc_custom!ZN15wxEventLoopBase3RunEv+0x137
0028f258 0c354cf0 181a7ae4 19e59250 00000000 wxmsw30ud_gcc_custom!ZN17wxDialogModalData7RunLoopEv+0x13
0028f2f8 004074e1 17d86654 0028f548 0028f3a0 wxmsw30ud_gcc_custom!ZN8wxDialog9ShowModalEv+0x18e
0028f598 00405156 181421c0 19c9de70 00000000 codeblocks_exe+0x74e1
0028fd28 0053b982 00000000 0028fd6c 00000000 codeblocks_exe+0x5156
0028fd48 0c1d1453 00000008 17d5d9f0 00027eb4 codeblocks_exe+0x13b982
0028fdc8 0c290d0c 0ce0d340 17d5d9f0 0cb1c7c8 wxmsw30ud_gcc_custom!Z11wxEntryRealRiPPw+0xab
0028fde8 0c2912c4 0ce0d340 17d5d9f0 0cb1c7c8 wxmsw30ud_gcc_custom!Z7wxEntryRiPPw+0x18
0028fe78 00402471 00400000 00000000 00000000 wxmsw30ud_gcc_custom!Z7wxEntryP11HINSTANCE__S0_Pci+0xd8
0028fe98 005ee2bb 00400000 00000000 031a5680 codeblocks_exe+0x2471
0028ff18 004010fd 7efde000 00000000 7efde000 codeblocks_exe+0x1ee2bb
0028ff94 773c9832 7efde000 77988401 00000000 codeblocks_exe+0x10fd
0028ffd4 773c9805 004012a0 7efde000 00000000 ntdll!__RtlUserThreadStart+0x70
0028ffec 00000000 004012a0 7efde000 00000000 ntdll!_RtlUserThreadStart+0x1b


STACK_COMMAND:  ~0s ; .cxr ; kb

THREAD_SHA1_HASH_MOD:  7d8c2ec9f442f84c8faab4b784f4842c9f2737ba

FOLLOWUP_IP:
wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26
0ca4dfd6 83ec10          sub     esp,10h

FAULT_INSTR_CODE:  c910ec83

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: wxmsw30ud_gcc_custom

IMAGE_NAME:  wxmsw30ud_gcc_custom.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  0

BUCKET_ID:  APPLICATION_HANG_wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26

FAILURE_EXCEPTION_CODE:  cfffffff

FAILURE_IMAGE_NAME:  wxmsw30ud_gcc_custom.dll

BUCKET_ID_IMAGE_STR:  wxmsw30ud_gcc_custom.dll

FAILURE_MODULE_NAME:  wxmsw30ud_gcc_custom

BUCKET_ID_MODULE_STR:  wxmsw30ud_gcc_custom

FAILURE_FUNCTION_NAME:  Z10GetMessageP6tagMSGP6HWND__jj

BUCKET_ID_FUNCTION_STR:  Z10GetMessageP6tagMSGP6HWND__jj

BUCKET_ID_OFFSET:  26

BUCKET_ID_MODTIMEDATESTAMP:  0

BUCKET_ID_MODCHECKSUM:  ae87499

BUCKET_ID_MODVER_STR:  3.0.4.0

BUCKET_ID_PREFIX_STR:  APPLICATION_HANG_

FAILURE_PROBLEM_CLASS:  APPLICATION_HANG_wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26

FAILURE_SYMBOL_NAME:  wxmsw30ud_gcc_custom.dll!Z10GetMessageP6tagMSGP6HWND__jj

FAILURE_BUCKET_ID:  APPLICATION_HANG_cfffffff_wxmsw30ud_gcc_custom.dll!Z10GetMessageP6tagMSGP6HWND__jj

TARGET_TIME:  2018-09-27T13:36:14.000Z


The problem is, i have no debug information (symbols ecc) for all this that work with windbg...

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #16 on: September 27, 2018, 04:43:28 pm »
I can also confirm that this happens within codeblocks, without patch building and on wx2.8

this seems to be some wxExecute thing...

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #17 on: September 27, 2018, 06:05:08 pm »
I think i have fixed this problem. Can you try this patch?

Basically it uses the functionality present in codeblocks, but not used/ deactivated. There is no where written why it is deactivated/not used...

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #18 on: September 27, 2018, 07:57:52 pm »
Have you tried git/svn blame?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #19 on: September 27, 2018, 09:11:36 pm »
quite old....
Code
commit b4ffe10e14c8c1417f89ef671cccc53efa0478f4
Author: Yiannis Mandravellos <mandrav@codeblocks.org>
Date:   Sun May 9 11:51:42 2004 +0000

    Initial import of project


    git-svn-id: https://svn.code.sf.net/p/codeblocks/code/trunk@5 2a5c6006-c6dd-42ca-98ab-0921f2732cef

diff --git a/src/sdk/pipedprocess.cpp b/src/sdk/pipedprocess.cpp
--- /dev/null
+++ b/src/sdk/pipedprocess.cpp
@@ -0,0 +55,22 @@
+{
+    m_Pid = wxExecute(cmd, wxEXEC_ASYNC, this);
+       if (m_Pid)
+       {
+//             m_timerPollProcess.SetOwner(this, idTimerPollProcess);
+//             m_timerPollProcess.Start(pollingInterval);
+       }
+       return m_Pid;
+}
+
+void PipedProcess::SendString(const wxString& text)
+{
+       //Manager::Get()->GetMessageManager()->Log(m_PageIndex, cmd);
+       wxOutputStream* pOut = GetOutputStream();
+       if (pOut)
+       {
+               wxTextOutputStream sin(*pOut);
+               wxString msg = text + '\n';
+               sin.WriteString(msg);
+       }
+}
+
(END)

Note: my patch is only for testing. The timer is not stopped after the process terminated... there is still some work to do, but first i would like know if this fixes the problem for op
« Last Edit: September 28, 2018, 10:49:29 am by BlueHazzard »

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #20 on: September 28, 2018, 10:29:15 am »
Great. I can confirm that the patch solves the problem for the small examples with/without using the fake_gcc. I am currently compiling the projects where I originally observed the problem in a Jenkins environment with various compilers. It's looking very good so far...

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #21 on: October 13, 2018, 11:00:24 am »
I can confirm now that the patch fixes the problem. I am able to batch build my projects with various compilers unattendedly on my Desktop and also on a Jenkins CI server. Thank's a lot for the support and the submitted fix. Please let me know if there is anything I can do to help with further steps, especially anything that would help towards integrating a fix for the next CB release.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #22 on: October 13, 2018, 11:42:13 am »
Can you try this patch (remove the old one):
Code
diff --git a/src/src/main.cpp b/src/src/main.cpp
index 886eead0f..7d4b74cc6 100644
--- a/src/src/main.cpp
+++ b/src/src/main.cpp
@@ -2720,7 +2720,6 @@ void MainFrame::OnApplicationClose(wxCloseEvent& event)

     CodeBlocksEvent evt(cbEVT_APP_START_SHUTDOWN);
     Manager::Get()->ProcessEvent(evt);
-    Manager::Yield();

     m_InitiatedShutdown = true;
     Manager::BlockYields(true);

The other patch just works around the problem. This one tries to fix it.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #23 on: October 15, 2018, 02:09:09 pm »
Can you explain why a change in the shut down code should help against an pipe/idle/flush problem happening in the middle of the application? The shutdown function is never called on the way to the bug, so why should it make any difference?

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #24 on: October 15, 2018, 02:51:21 pm »
Calling wxWindow::Close adds some stuff to be cleared on the next idle (PendingDelete or something).
Closing the frame calls the OnApplicationClose method and when Yield is called the PendingDelete stuff is executed too early.
Some code after the Yield uses controls destroyed in PendingDelete. If you check my comment in the ticket you'll see the backtraces.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #25 on: October 15, 2018, 02:52:50 pm »
But I guess, I've posted on the wrong ticket... I thought this is the crash-on-exit issue.

Sorry...
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Miguel Gimenez

  • Developer
  • Lives here!
  • *****
  • Posts: 1553
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #26 on: October 15, 2018, 04:39:56 pm »
@obfuscated, the crash issue was indeed at the beginning of this thread, later moved to ticket 738. I have tested your patch, and the crash is gone. No apparent side effects neither in batch mode nor in GUI mode (Windows 7).

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #27 on: July 18, 2019, 10:46:30 am »
To come back to this, because the stall hits me at the moment...
the patch obfuscated posted here in
Can you try this patch (remove the old one):
Code
diff --git a/src/src/main.cpp b/src/src/main.cpp
index 886eead0f..7d4b74cc6 100644
--- a/src/src/main.cpp
+++ b/src/src/main.cpp
@@ -2720,7 +2720,6 @@ void MainFrame::OnApplicationClose(wxCloseEvent& event)

     CodeBlocksEvent evt(cbEVT_APP_START_SHUTDOWN);
     Manager::Get()->ProcessEvent(evt);
-    Manager::Yield();

     m_InitiatedShutdown = true;
     Manager::BlockYields(true);

The other patch just works around the problem. This one tries to fix it.

is not for the stalling but for a crash? Am i right?
The stalling is still a problem also with this patch?

[Edit:] i also created a ticket about the stall: https://sourceforge.net/p/codeblocks/tickets/854/

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #28 on: July 18, 2019, 10:51:23 am »
@simons
If you are still here: Can you give some example how you use codeblocks with Jenkins? Some minimal configuration file and some use cases?
 Thank you!

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #29 on: July 18, 2019, 05:09:41 pm »
@BlueHazzard: Yes, still here. Thank's for opening a ticket. The patch from @oBFusCATed did not solve the stall problem for me, but your patch posted before that did (test_FixCompilingIdleHanging_use_PipedProcessLaunch_1.patch). I've running a patched version of CodeBlocks for a few months now on my server.

About my Jenkins setup: I run codeblocks from within a Jenkins pipeline through a batch file.

The pipeline looks something like this:
Code
...
stage('Build'){
  steps {
    bat "run_codeblocks.bat my_cb_workspace.workspace --build --target=release"
  }
}
...

and the batch file 'run_codeblocks.bat' runs CodeBlocks like this:
Code
...
codeblocks.exe" --no-check-associations --no-dde --no-splash-screen %*
...

One could probably just run CodeBlocks directly from the 'bat'-step in the pipeline. However, we do some extra steps in the batch file to work-around some Visual Studio specific issues (like mspdbsrv.exe being killed by Jenkins when building debug targets).

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #30 on: July 18, 2019, 07:54:14 pm »
is not for the stalling but for a crash? Am i right?
The stalling is still a problem also with this patch?
Yep, it is fixing a crash. I though I've pushed it, is this the case?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #31 on: July 19, 2019, 10:06:45 am »
@simons: thank you! You are running this only on windows?

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #32 on: July 19, 2019, 11:59:50 am »
Yes, windows only.