Author Topic: Batch build problems (hangs/stalls if log window is not active)  (Read 17427 times)

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #15 on: September 27, 2018, 03:53:49 pm »
Has anyone been able to reproduce the problem using the "fake_gcc"? I pushed the new configuration to the repo. Any ideas on how to move on from here?
no, i was not able to reproduce it with your latest fake_gcc...

I have tried future with WinDBG and gcc and i got this information:
cc1plus.exe:
Code
DEFAULT_BUCKET_ID:  APPLICATION_HANG_BlockedOn_FileIO

PRIMARY_PROBLEM_CLASS:  BlockedOn_FileIO

THREAD_SHA1_HASH_MOD_FUNC:  c281b8792a9b0b7b09f56f4b37bca845ce1f1b92

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  5428a93740841021c6fded59898de991617bd52a

LAST_CONTROL_TRANSFER:  from 7618e004 to 773af96d

STACK_TEXT: 
0028f340 7618e004 00000044 00000000 00000000 ntdll!NtWriteFile+0x15
0028f3a4 76ca12cc 00000044 0028f418 00000400 KERNELBASE!WriteFile+0x113
0028f3c0 76dd4236 00000044 0028f418 00000400 kernel32!WriteFileImplementation+0x76
0028f97c 76dd40eb 00000002 12b99598 00000877 msvcrt!_write_nolock+0x3fb
0028f9c0 76dcf15e 00000002 12b99598 00000877 msvcrt!_write+0x9f
0028f9e0 76dcf1a0 76e62940 00000000 0028fa28 msvcrt!_flush+0x3b
0028f9f0 76dd416d 76e62940 ff7a9eca 01024a9c msvcrt!_fflush_nolock+0x1c
0028fa28 00ea85fd 76e62940 0028fb18 0028fb18 msvcrt!fflush+0x30
WARNING: Stack unwind information not available. Following frames may be wrong.
0028faf8 00ea505e 01463500 0028fb18 00000034 cc1plus+0xaa85fd
0028fb48 00429b48 000001aa 01024a9c 15d3a2c0 cc1plus+0xaa505e
0028fb58 0043f039 15153910 00000000 01bc5540 cc1plus+0x29b48
0028fbb8 004cc7c0 00000001 00000001 00000000 cc1plus+0x3f039
0028fbd8 004cd4f1 1917b708 00465250 0c95bf80 cc1plus+0xcc7c0
0028fbf8 0043fd61 1917b708 00000000 016b9cc0 cc1plus+0xcd4f1
00000000 00000000 00000000 00000000 00000000 cc1plus+0x3fd61

mingw32g++.exe:
Code
DERIVED_WAIT_CHAIN:    00 ca8.24c8 Process Handle 

WAIT_CHAIN_COMMAND:  ~0s;k;;

THREAD_ATTRIBUTES:
BLOCKING_THREAD:  000024c8

DEFAULT_BUCKET_ID:  APPLICATION_HANG_BlockedOn_ProcessHandle

PRIMARY_PROBLEM_CLASS:  BlockedOn_ProcessHandle

THREAD_SHA1_HASH_MOD_FUNC:  e48a380cac1320f60a2487066e7609815690d73a

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  bcd69aa1f032d972611d5e1bfe1a686de0b4cd74

LAST_CONTROL_TRANSFER:  from 761915ce to 773af901

STACK_TEXT: 
0028f490 761915ce 00000054 00000000 00000000 ntdll!NtWaitForSingleObject+0x15
0028f4fc 76ca1194 00000054 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
0028f514 76ca1148 00000054 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
0028f528 0044c64b 00000054 ffffffff 0028f6a8 kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0028f538 00444aa8 00000004 00360100 0036fe33 mingw32_g__+0x4c64b
0028f6a8 0040a4c5 0036c368 004f4220 0000002f mingw32_g__+0x44aa8
0028f738 0040ca8e 003688f9 004f4aa3 00000001 mingw32_g__+0xa4c5
0028f79c 773c3476 77981bc8 0047fe4a 004f4aa1 mingw32_g__+0xca8e
0028f938 00401577 004f3470 0047ff78 00000019 ntdll!RtlpAllocateHeap+0xe66
0028f958 0040a917 0047c9b4 00361c32 00000009 mingw32_g__+0x1577
0028f9e8 0040ca8e 003688f9 004f3fdb 0000000c mingw32_g__+0xa917
0028fa48 00406a05 773c3476 77981898 004f3bcd mingw32_g__+0xca8e
0028faf8 0040ca8e 003688f9 004f3d3b 00000002 mingw32_g__+0x6a05
0028fb5c 773c3476 77981f88 0047db8d 004f3d39 mingw32_g__+0xca8e
0028fc88 0040b295 00000002 00000002 773c4355 ntdll!RtlpAllocateHeap+0xe66
0028fd18 0040ca8e 003688f9 0047db8b 00000001 mingw32_g__+0xb295
0028fd74 773c3476 003c34a1 00000001 00000010 mingw32_g__+0xca8e
0028fe48 0040e0b0 00000000 00000000 004f1a58 ntdll!RtlpAllocateHeap+0xe66
0028fe78 0040e1d8 0047db48 00000001 0028ffc4 mingw32_g__+0xe0b0
0028ff18 004010fd 0028ff28 76dd9e34 fffde000 mingw32_g__+0xe1d8
0028ff94 773c9832 fffde000 77981c68 00000000 mingw32_g__+0x10fd
0028ffd4 773c9805 00401280 fffde000 00000000 ntdll!__RtlUserThreadStart+0x70
0028ffec 00000000 00401280 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b

codeblocks.exe:
Code
DERIVED_WAIT_CHAIN:    00 26b8.1fbc Unknown 

WAIT_CHAIN_COMMAND:  ~0s;k;;

THREAD_ATTRIBUTES:
BLOCKING_THREAD:  00001fbc

THREAD_SHA1_HASH_MOD_FUNC:  71431138aca2c5693a1cab1a8c383575c7f4d911

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  ca61d71375c8d331e524a6f7a93b8a1e35ccd897

LAST_CONTROL_TRANSFER:  from 76ea790d to 76ea78d7

STACK_TEXT: 
0028eefc 76ea790d 0028f0b8 00000000 00000000 user32!NtUserGetMessage+0x15
0028ef18 0ca4dfd6 0028f0b8 00000000 00000000 user32!GetMessageW+0x33
WARNING: Stack unwind information not available. Following frames may be wrong.
0028ef38 0c24f8a3 0028f0b8 00000000 00000000 wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+0x26
0028f048 0c35a7aa 0028f0b8 0028f138 0028f068 wxmsw30ud_gcc_custom!ZN18wxMSWEventLoopBase14GetNextMessageEP6tagMSG+0x77
0028f0f8 0c1947e9 00000000 0108fb44 0000032c wxmsw30ud_gcc_custom!ZN14wxGUIEventLoop8DispatchEv+0x52
0028f118 0c1948b1 0028f138 00001fbc 00000000 wxmsw30ud_gcc_custom!ZN17wxEventLoopManual13ProcessEventsEv+0x4d
0028f198 0c194503 0028f210 19e5925c 00000000 wxmsw30ud_gcc_custom!ZN17wxEventLoopManual5DoRunEv+0xc5
0028f238 0cac3ad3 00000024 0028f2cc 0028f258 wxmsw30ud_gcc_custom!ZN15wxEventLoopBase3RunEv+0x137
0028f258 0c354cf0 181a7ae4 19e59250 00000000 wxmsw30ud_gcc_custom!ZN17wxDialogModalData7RunLoopEv+0x13
0028f2f8 004074e1 17d86654 0028f548 0028f3a0 wxmsw30ud_gcc_custom!ZN8wxDialog9ShowModalEv+0x18e
0028f598 00405156 181421c0 19c9de70 00000000 codeblocks_exe+0x74e1
0028fd28 0053b982 00000000 0028fd6c 00000000 codeblocks_exe+0x5156
0028fd48 0c1d1453 00000008 17d5d9f0 00027eb4 codeblocks_exe+0x13b982
0028fdc8 0c290d0c 0ce0d340 17d5d9f0 0cb1c7c8 wxmsw30ud_gcc_custom!Z11wxEntryRealRiPPw+0xab
0028fde8 0c2912c4 0ce0d340 17d5d9f0 0cb1c7c8 wxmsw30ud_gcc_custom!Z7wxEntryRiPPw+0x18
0028fe78 00402471 00400000 00000000 00000000 wxmsw30ud_gcc_custom!Z7wxEntryP11HINSTANCE__S0_Pci+0xd8
0028fe98 005ee2bb 00400000 00000000 031a5680 codeblocks_exe+0x2471
0028ff18 004010fd 7efde000 00000000 7efde000 codeblocks_exe+0x1ee2bb
0028ff94 773c9832 7efde000 77988401 00000000 codeblocks_exe+0x10fd
0028ffd4 773c9805 004012a0 7efde000 00000000 ntdll!__RtlUserThreadStart+0x70
0028ffec 00000000 004012a0 7efde000 00000000 ntdll!_RtlUserThreadStart+0x1b


STACK_COMMAND:  ~0s ; .cxr ; kb

THREAD_SHA1_HASH_MOD:  7d8c2ec9f442f84c8faab4b784f4842c9f2737ba

FOLLOWUP_IP:
wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26
0ca4dfd6 83ec10          sub     esp,10h

FAULT_INSTR_CODE:  c910ec83

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: wxmsw30ud_gcc_custom

IMAGE_NAME:  wxmsw30ud_gcc_custom.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  0

BUCKET_ID:  APPLICATION_HANG_wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26

FAILURE_EXCEPTION_CODE:  cfffffff

FAILURE_IMAGE_NAME:  wxmsw30ud_gcc_custom.dll

BUCKET_ID_IMAGE_STR:  wxmsw30ud_gcc_custom.dll

FAILURE_MODULE_NAME:  wxmsw30ud_gcc_custom

BUCKET_ID_MODULE_STR:  wxmsw30ud_gcc_custom

FAILURE_FUNCTION_NAME:  Z10GetMessageP6tagMSGP6HWND__jj

BUCKET_ID_FUNCTION_STR:  Z10GetMessageP6tagMSGP6HWND__jj

BUCKET_ID_OFFSET:  26

BUCKET_ID_MODTIMEDATESTAMP:  0

BUCKET_ID_MODCHECKSUM:  ae87499

BUCKET_ID_MODVER_STR:  3.0.4.0

BUCKET_ID_PREFIX_STR:  APPLICATION_HANG_

FAILURE_PROBLEM_CLASS:  APPLICATION_HANG_wxmsw30ud_gcc_custom!Z10GetMessageP6tagMSGP6HWND__jj+26

FAILURE_SYMBOL_NAME:  wxmsw30ud_gcc_custom.dll!Z10GetMessageP6tagMSGP6HWND__jj

FAILURE_BUCKET_ID:  APPLICATION_HANG_cfffffff_wxmsw30ud_gcc_custom.dll!Z10GetMessageP6tagMSGP6HWND__jj

TARGET_TIME:  2018-09-27T13:36:14.000Z


The problem is, i have no debug information (symbols ecc) for all this that work with windbg...

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #16 on: September 27, 2018, 04:43:28 pm »
I can also confirm that this happens within codeblocks, without patch building and on wx2.8

this seems to be some wxExecute thing...

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #17 on: September 27, 2018, 06:05:08 pm »
I think i have fixed this problem. Can you try this patch?

Basically it uses the functionality present in codeblocks, but not used/ deactivated. There is no where written why it is deactivated/not used...

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #18 on: September 27, 2018, 07:57:52 pm »
Have you tried git/svn blame?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #19 on: September 27, 2018, 09:11:36 pm »
quite old....
Code
commit b4ffe10e14c8c1417f89ef671cccc53efa0478f4
Author: Yiannis Mandravellos <mandrav@codeblocks.org>
Date:   Sun May 9 11:51:42 2004 +0000

    Initial import of project


    git-svn-id: https://svn.code.sf.net/p/codeblocks/code/trunk@5 2a5c6006-c6dd-42ca-98ab-0921f2732cef

diff --git a/src/sdk/pipedprocess.cpp b/src/sdk/pipedprocess.cpp
--- /dev/null
+++ b/src/sdk/pipedprocess.cpp
@@ -0,0 +55,22 @@
+{
+    m_Pid = wxExecute(cmd, wxEXEC_ASYNC, this);
+       if (m_Pid)
+       {
+//             m_timerPollProcess.SetOwner(this, idTimerPollProcess);
+//             m_timerPollProcess.Start(pollingInterval);
+       }
+       return m_Pid;
+}
+
+void PipedProcess::SendString(const wxString& text)
+{
+       //Manager::Get()->GetMessageManager()->Log(m_PageIndex, cmd);
+       wxOutputStream* pOut = GetOutputStream();
+       if (pOut)
+       {
+               wxTextOutputStream sin(*pOut);
+               wxString msg = text + '\n';
+               sin.WriteString(msg);
+       }
+}
+
(END)

Note: my patch is only for testing. The timer is not stopped after the process terminated... there is still some work to do, but first i would like know if this fixes the problem for op
« Last Edit: September 28, 2018, 10:49:29 am by BlueHazzard »

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #20 on: September 28, 2018, 10:29:15 am »
Great. I can confirm that the patch solves the problem for the small examples with/without using the fake_gcc. I am currently compiling the projects where I originally observed the problem in a Jenkins environment with various compilers. It's looking very good so far...

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #21 on: October 13, 2018, 11:00:24 am »
I can confirm now that the patch fixes the problem. I am able to batch build my projects with various compilers unattendedly on my Desktop and also on a Jenkins CI server. Thank's a lot for the support and the submitted fix. Please let me know if there is anything I can do to help with further steps, especially anything that would help towards integrating a fix for the next CB release.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #22 on: October 13, 2018, 11:42:13 am »
Can you try this patch (remove the old one):
Code
diff --git a/src/src/main.cpp b/src/src/main.cpp
index 886eead0f..7d4b74cc6 100644
--- a/src/src/main.cpp
+++ b/src/src/main.cpp
@@ -2720,7 +2720,6 @@ void MainFrame::OnApplicationClose(wxCloseEvent& event)

     CodeBlocksEvent evt(cbEVT_APP_START_SHUTDOWN);
     Manager::Get()->ProcessEvent(evt);
-    Manager::Yield();

     m_InitiatedShutdown = true;
     Manager::BlockYields(true);

The other patch just works around the problem. This one tries to fix it.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #23 on: October 15, 2018, 02:09:09 pm »
Can you explain why a change in the shut down code should help against an pipe/idle/flush problem happening in the middle of the application? The shutdown function is never called on the way to the bug, so why should it make any difference?

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #24 on: October 15, 2018, 02:51:21 pm »
Calling wxWindow::Close adds some stuff to be cleared on the next idle (PendingDelete or something).
Closing the frame calls the OnApplicationClose method and when Yield is called the PendingDelete stuff is executed too early.
Some code after the Yield uses controls destroyed in PendingDelete. If you check my comment in the ticket you'll see the backtraces.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #25 on: October 15, 2018, 02:52:50 pm »
But I guess, I've posted on the wrong ticket... I thought this is the crash-on-exit issue.

Sorry...
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Miguel Gimenez

  • Developer
  • Lives here!
  • *****
  • Posts: 1553
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #26 on: October 15, 2018, 04:39:56 pm »
@obfuscated, the crash issue was indeed at the beginning of this thread, later moved to ticket 738. I have tested your patch, and the crash is gone. No apparent side effects neither in batch mode nor in GUI mode (Windows 7).

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #27 on: July 18, 2019, 10:46:30 am »
To come back to this, because the stall hits me at the moment...
the patch obfuscated posted here in
Can you try this patch (remove the old one):
Code
diff --git a/src/src/main.cpp b/src/src/main.cpp
index 886eead0f..7d4b74cc6 100644
--- a/src/src/main.cpp
+++ b/src/src/main.cpp
@@ -2720,7 +2720,6 @@ void MainFrame::OnApplicationClose(wxCloseEvent& event)

     CodeBlocksEvent evt(cbEVT_APP_START_SHUTDOWN);
     Manager::Get()->ProcessEvent(evt);
-    Manager::Yield();

     m_InitiatedShutdown = true;
     Manager::BlockYields(true);

The other patch just works around the problem. This one tries to fix it.

is not for the stalling but for a crash? Am i right?
The stalling is still a problem also with this patch?

[Edit:] i also created a ticket about the stall: https://sourceforge.net/p/codeblocks/tickets/854/

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #28 on: July 18, 2019, 10:51:23 am »
@simons
If you are still here: Can you give some example how you use codeblocks with Jenkins? Some minimal configuration file and some use cases?
 Thank you!

Offline simons

  • Multiple posting newcomer
  • *
  • Posts: 10
Re: Batch build problems (hangs/stalls if log window is not active)
« Reply #29 on: July 18, 2019, 05:09:41 pm »
@BlueHazzard: Yes, still here. Thank's for opening a ticket. The patch from @oBFusCATed did not solve the stall problem for me, but your patch posted before that did (test_FixCompilingIdleHanging_use_PipedProcessLaunch_1.patch). I've running a patched version of CodeBlocks for a few months now on my server.

About my Jenkins setup: I run codeblocks from within a Jenkins pipeline through a batch file.

The pipeline looks something like this:
Code
...
stage('Build'){
  steps {
    bat "run_codeblocks.bat my_cb_workspace.workspace --build --target=release"
  }
}
...

and the batch file 'run_codeblocks.bat' runs CodeBlocks like this:
Code
...
codeblocks.exe" --no-check-associations --no-dde --no-splash-screen %*
...

One could probably just run CodeBlocks directly from the 'bat'-step in the pipeline. However, we do some extra steps in the batch file to work-around some Visual Studio specific issues (like mspdbsrv.exe being killed by Jenkins when building debug targets).