Author Topic: Batch build problems (hangs/stalls if log window is not active)  (Read 17425 times)

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Batch build problems (hangs/stalls if log window is not active)
« on: September 18, 2018, 05:33:03 pm »
Hi all,

I am trying to use the batch build feature of Codeblocks to have my projects built automatically on a Jenkins CI server. This may be related to some earlier discussions:
- http://forums.codeblocks.org/index.php/topic,21838.msg148541.html#msg148541 (Linux and assumingly outdated solution "special build by Jens")
- http://forums.codeblocks.org/index.php/topic,14928.0.html (Linux solution: export DISPLAY=:2000.0)
- http://forums.codeblocks.org/index.php?topic=15662.0 (codeblocks_console branch, found it on github, assumingly outdated, could not get it to compile)

The problem that I get is on MS Windows 7 and with projects that produce *a lot* of compiler output (template-heavy code). Here is what happens:
- Running codeblocks with a command like this: codeblocks.exe <PATH_TO_WORKSPACE> --rebuild --no-check-associations --no-dde --no-splash-screen --no-batch-window-close --target=<TARGET>
- The batch build window opens and the build starts (I can see compiler output in the log window and cl.exe or mingw-gcc.exe shows up in the task manager's process list consuming cpu time and memory)
- After a while of not activating the batch log window, the build stalls, there is no more compiler output being appended and the compiler executables cpu usage goes down to zero
- Only if I hover over or click on the batch log window, the build continues, I can see new compiler output and cpu time being consumed
- Leaving the batch log window minimized or just not focused, the build stalls again

I already tried a couple of things:
- I looked into the Codeblocks code trying to find an easy way to "deactivate" the batch log window.. my current conclusion is that there is none ;)
- I managed to implement a hack that disables printing all the compiler output into the log window (remember the problem seems to only occur with projects that produce *much* output). The window is empty now, but the problem persists
- I tried to get a backtrace using gdb, which I don't know very well so I'm not sure if I did it right. After starting up gdb with codeblocks.exe as the debugee and waiting for the build to stall, I had to use a little helper application (http://www.mingw.org/wiki/Workaround_for_GDB_Ctrl_C_Interrupt) to interrupt the process and to be able to get a backtrace

Here is the result of "(gdb) thread apply all bt":
Code
(gdb) thread apply all bt

Thread 146 (Thread 6524.0x1b84):
#0  0x77d5000d in ntdll!DbgBreakPoint () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77ddf306 in ntdll!DbgUiRemoteBreakin () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x3e10b887 in ?? ()
#3  0x00000000 in ?? ()

Thread 145 (Thread 6524.0x1ac4):
#0  0x77d61f76 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d61f76 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x77d8fa8e in ntdll!TpSetTimer () from C:\WINDOWS\SysWOW64\ntdll.dll
#3  0x7780343d in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\syswow64\kernel32.dll
#4  0x77d79832 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#5  0x77d79805 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#6  0x00000000 in ?? ()

Thread 143 (Thread 6524.0x1cec):
#0  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x76ed15ce in WaitForSingleObjectEx () from C:\WINDOWS\syswow64\KernelBase.dll
#3  0x000001b0 in ?? ()
#4  0x00000000 in ?? ()

Thread 51 (Thread 6524.0x22b0):
---Type <return> to continue, or q <return> to quit---
#0  0x77d61f76 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d61f76 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x77d8fa8e in ntdll!TpSetTimer () from C:\WINDOWS\SysWOW64\ntdll.dll
#3  0x7780343d in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\syswow64\kernel32.dll
#4  0x77d79832 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#5  0x77d79805 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#6  0x00000000 in ?? ()

Thread 47 (Thread 6524.0x9c4):
#0  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x76ed15ce in WaitForSingleObjectEx () from C:\WINDOWS\syswow64\KernelBase.dll
#3  0x00000290 in ?? ()
#4  0x00000000 in ?? ()

Thread 46 (Thread 6524.0x17f0):
#0  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x76ed15ce in WaitForSingleObjectEx () from C:\WINDOWS\syswow64\KernelBase.dll
#3  0x0000032c in ?? ()
#4  0x00000000 in ?? ()

Thread 45 (Thread 6524.0x86c):
#0  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
---Type <return> to continue, or q <return> to quit---
#1  0x77d5f901 in ntdll!ZwWaitForSingleObject () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x76ed15ce in WaitForSingleObjectEx () from C:\WINDOWS\syswow64\KernelBase.dll
#3  0x000001e8 in ?? ()
#4  0x00000000 in ?? ()

Thread 2 (Thread 6524.0x1670):
#0  0x77d6018d in ntdll!ZwWaitForMultipleObjects () from C:\WINDOWS\SysWOW64\ntdll.dll
#1  0x77d6018d in ntdll!ZwWaitForMultipleObjects () from C:\WINDOWS\SysWOW64\ntdll.dll
#2  0x77d8f69f in ntdll!_allmul () from C:\WINDOWS\SysWOW64\ntdll.dll
#3  0x00000003 in ?? ()
#4  0x014ecf50 in ?? ()
#5  0x7780343d in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\syswow64\kernel32.dll
#6  0x77d79832 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#7  0x77d79805 in ntdll!RtlInitializeExceptionChain () from C:\WINDOWS\SysWOW64\ntdll.dll
#8  0x00000000 in ?? ()

Thread 1 (Thread 6524.0x10f8):
#0  0x767578d7 in USER32!DispatchMessageW () from C:\WINDOWS\syswow64\user32.dll
#1  0x627df5cc in wxEventLoop::Dispatch() ()
   from D:\Users\scho_so\Dev\codeblocks-17.12-BatchFix\src\devel\wxmsw28u_gcc_custom.dll
#2  0x628b1798 in wxEventLoopManual::Run() ()
   from D:\Users\scho_so\Dev\codeblocks-17.12-BatchFix\src\devel\wxmsw28u_gcc_custom.dll
#3  0x0a9a9448 in ?? ()
#4  0x8775d038 in ?? ()
---Type <return> to continue, or q <return> to quit---
#5  0x3128c483 in ?? ()
#6  0x04c25bc0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Does anyone have ideas about ...
... where the problem with a stalling build log window could originate from (e.g. wxWidgets modal windows / event loops etc.) ?
... how to further identify the problem, e.g. using some more advanced debugging techniques?
... things to try in order to find a work-around (e.g. some 'hackish' solution until someone can get a grip of the actual problem) ?

Help is much appreciated here and I would be more than willing to try out possible solutions or debugging techniques. Unfortunately, I haven't been able to reproduce the problem in a sort of "minimal example" that I could share, since it only comes up with certain projects that produce a lot of template-tracing compiler outputs. That's also the reason, I haven't issued a bug report...

Thanks everyone.


Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #2 on: September 18, 2018, 06:39:07 pm »
simons: Can you try to disable the option in the cmd.exe's settings? If it works we'll add it to the code, so it is automatic.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #3 on: September 19, 2018, 09:17:59 am »
Thanks for the quick response.

I did not have the "quick edit mode" enabled in my command prompt. Anyway, I tried to enable/disable it without any changes to the symptoms.

Just to clarify: It's the Codeblocks batch build log window (a little wxWidgets modal dialog, as far as I read from the sources) that stalls and doesn't show any more compiler output. It's *not* the command prompt. Actually Codeblocks - even in batch mode - seems to completely detach from the command prompt such that I can close the command prompt window and just have the Codeblocks batch build log window open. It is this window that seems to stall the compile process when not focused for a while. Any GUI event, even hovering over the window with the mouse or dragging another window over it, will continue the build process. As soon as I put my hands off the mouse and the window does not have focus, it stalls again.

Any more ideas?

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #4 on: September 19, 2018, 12:06:07 pm »
It could be the idle handler we're probably using.
Can you reproduce it with an open source project?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Miguel Gimenez

  • Developer
  • Lives here!
  • *****
  • Posts: 1553
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #5 on: September 19, 2018, 02:26:00 pm »
Checking this issue I found another problem. After the reuild is finished, I close the window and a SIGSEGV is received:

Code
Thread 1 received signal SIGSEGV, Segmentation fault.
0x0046db13 in InfoPane::Show (this=0x8861a68, logger=0xbd81cd8) at G:\Codeblocks\src\src\infopane.cpp:253
253             if (m_Pages.Item(i)->logger == logger)
(gdb) bt
#0  0x0046db13 in InfoPane::Show (this=0x8861a68, logger=0xbd81cd8) at G:\Codeblocks\src\src\infopane.cpp:253
#1  0x00496b96 in MainFrame::OnSwitchToLogWindow (this=0x87a6670, event=...) at G:\Codeblocks\src\src\main.cpp:5036
#2  0x00513a3c in cbEventFunctor<MainFrame, CodeBlocksLogEvent>::Call (this=0x8845038, event=...)
    at G:/Codeblocks/src/include/cbfunctor.h:49
#3  0x01fe7833 in Manager::ProcessEvent (this=0x7ea7310, event=...) at G:\Codeblocks\src\sdk\manager.cpp:328
#4  0x6a157d8b in CompilerGCC::ClearLog (this=0x88f3878) at G:\Codeblocks\src\plugins\compilergcc\compilergcc.cpp:1061
#5  0x6a16ab0b in CompilerGCC::OnWorkspaceClosed (this=0x88f3878, event=...)
    at G:\Codeblocks\src\plugins\compilergcc\compilergcc.cpp:3405
#6  0x6a1d5b3c in cbEventFunctor<CompilerGCC, CodeBlocksEvent>::Call (this=0xa10f358, event=...)
    at G:/Codeblocks/src/include/cbfunctor.h:49
#7  0x01fe7581 in Manager::ProcessEvent (this=0x7ea7310, event=...) at G:\Codeblocks\src\sdk\manager.cpp:260
#8  0x02002507 in PluginManager::NotifyPlugins (this=0x88e16e0, event=...) at G:\Codeblocks\src\sdk\pluginmanager.cpp:1522
#9  0x020297ac in ProjectManager::CloseWorkspace (this=0x885ede8) at G:\Codeblocks\src\sdk\projectmanager.cpp:613
#10 0x0047fd78 in MainFrame::DoCloseCurrentWorkspace (this=0x87a6670) at G:\Codeblocks\src\src\main.cpp:1877
#11 0x0048945b in MainFrame::OnApplicationClose (this=0x87a6670, event=...) at G:\Codeblocks\src\src\main.cpp:2758
#12 0x644433d2 in wxAppConsoleBase::HandleEvent(wxEvtHandler*, void (wxEvtHandler::*)(wxEvent&), wxEvent&) const ()
   from C:\Windows\system32\wxmsw311u_gcc_custom.dll
#13 0x0028f5f0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) print i
$1 = 0
(gdb) print m_Pages
$2 = {<wxBaseArrayPtrVoid> = {m_nSize = 4277075694, m_nCount = 4277075694, m_pItems = 0xfeeefeee}, <No data fields>}

I think m_Pages has not been initialized at this point. 4277075694 = 0xfeeefeee.

EDIT: the attached patch fixes this, I'll create a ticket.
« Last Edit: September 19, 2018, 02:46:05 pm by Miguel Gimenez »

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #6 on: September 19, 2018, 05:09:42 pm »
I managed to assemble a "somewhat minimal" example using boost::msm (template-heavy code) that reproduces the problem when compiling with visual studio 2005 (msvc8) or visual studio 2010 (msvc10).

It seems like there are a couple conditions that raise the probability of the problem: many compiler warnings and errors, multiple compiler processes, a decent cpu load on the compiler. I also played around with some code snippets from https://codegolf.stackexchange.com/questions/5802/write-the-shortest-program-that-generates-the-most-compiler-warnings-and-errors and while I managed to totally hog the Codeblocks application with some of these "compiler bombs", I could not reproduce my original problem. Also, bare in mind that I have seen the issue with only a single compiler process as well.

I uploaded the "not-so-minimal" example here: https://github.com/simfinite/test_codeblocks_batch_build
Please, follow the steps in the README to reproduce the problem (and sorry for making you download the boost sources...  ;))

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #7 on: September 19, 2018, 05:45:07 pm »
Can you reproduce this with mingw? The older versions of Visual studio aren't made to run in parallel.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #8 on: September 19, 2018, 10:47:11 pm »
I can reproduce this on my machine with windows7 mingw-tdm-5 and the example project... Funny...
For me it always stops at linking stage. I am the only one?

I can not reproduce the SIGSEGV from Miguel Gimenez

There really should be a codeblocks console builder... It is nearly a shame that this does not exists..

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #9 on: September 20, 2018, 10:36:32 am »
Quote
I can reproduce this on my machine with windows7 mingw-tdm-5

Interesting, I could not reproduce it with gcc on my machine yesterday. But I added a little warning generator to the example code and now I can reproduce the problem with
- msvc8
- msvc10
- TDM gcc 5.1.0
- MinGW-W64 gcc 5.4.0
(all with 4 build jobs)

The problem is still most easily reproduced with the old and weak visual studio compilers and hardest with gcc 5.4.0. It may take some tweaking of the amount of warnings generated (see fsm.hpp, line #206) or the number of build jobs. Although this sounds like an undeterministic problem, I could reproduce it *every time* once I found a configuration that works.


Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #10 on: September 20, 2018, 12:17:46 pm »
i would like to add, that i have enabled -wall and -wpedantic to generate even more messages

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #11 on: September 20, 2018, 06:02:12 pm »
Can you write a simple app that just spits output which looks like warnings generated by GCC and set this as the compiler executable?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #12 on: September 21, 2018, 12:16:02 am »
Can you write a simple app that just spits output which looks like warnings generated by GCC and set this as the compiler executable?
tried it right now with > 10000 warnings and i am not able to trigger this. The output count should be a lot more then from the working project. At least it needs a lot more time to ouput.. There has to be something else contributing to this...

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #13 on: September 21, 2018, 09:54:07 am »
Quote
tried it right now with > 10000 warnings and i am not able to trigger this
Yes, my observation is that the problem occurs only for the combination of many warnings being produced *and* heavy CPU load from the compiler processes. That's the reason I included boost::msm into the example application, because the simple warning generator wasn't enough and compiling the template-heavy library code consumes a lot of CPU time. The sample application pretty much reproduces the setting that I have in the projects where I originally observed the problem.

Quote
Can you write a simple app that just spits output which looks like warnings generated by GCC and set this as the compiler executable?
Yes, I wrote a little "fake_gcc" and put it into the same repository. I was able to reproduce the problem with it, however, some tweaking may still be required. There are some variables inside the main.cpp that configure how the program behaves. They are set to what worked on my machine. If you are having trouble reproducing the problem, try to increase/decrease "work_us" to increase/decrease CPU time consumption and check that the processes are in fact consuming considerable CPU time (e.g. 15-25% each with 4 build jobs on a quad core).

One thing I noticed using the "fake_gcc": The problem seems to only arise when compiler processes start up and/or quit. For example, I could not reproduce the problem just letting fake_gcc run forever. The project being built also needs to have many compilation units, such that new compiler processes are started up every now and then.
« Last Edit: September 21, 2018, 12:59:49 pm by simons »

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #14 on: September 27, 2018, 09:28:11 am »
I did some more experiments and have some minor insights to share:

- As I mentioned before, up until now I could only reproduce the problem if the compiler process is consuming considerable CPU time. I verified that the problem cannot be reproduced by hogging the CPU with some other processes and letting the compiler process output warnings only.

- However, now I found a configuration for the "fake_gcc"  that reproduces the problem without the busy loop, i.e. without consuming CPU time and by but just letting the compiler process sleep for a while each iteration. It's a bit harder to reproduce than the variant with the busy loop and only works with 4 build jobs on my machine. Still this decouples the problem from CPU usage and is pointing more towards a general timing issue.

Has anyone been able to reproduce the problem using the "fake_gcc"? I pushed the new configuration to the repo. Any ideas on how to move on from here?