Author Topic: Deadlock of C::B in CodeCompletion  (Read 869 times)

Offline Bat

  • Multiple posting newcomer
  • *
  • Posts: 47
Deadlock of C::B in CodeCompletion
« on: June 13, 2017, 11:25:33 am »
My configuration : custom C::B build (quite some modification), 32 bits, Win7, wx2.8.12.
I can modify/rebuild wx and C::B

I randomly (*) end in a deadlock, C::B unresponsive, not eating any processor
(*)can be 5 in a row, just after opening C::B or last for one week to appear. Seems to be related to activity - the more agressive I am on keyboard, the more it seems to occurs. I've no easy reproducable steps.
I would nearly say that it has nearly always occured, it's possible I see it more often now.

So to debug this, I launch C::B in GDB and work with that [quite a bit slow, on codecompletion specifically]
I share my findings :

So I just deadlockeded(*), I break process in GDB :
(*) It's not my first deadlock trace, and all are consistent with following trace

main thread stack trace
Quote
Code: [Select]
#0 0x7790f8b1 ntdll!ZwWaitForSingleObject() (C:\Windows\SysWOW64\ntdll.dll:??)
#1 0x7790f8b1 ntdll!ZwWaitForSingleObject() (C:\Windows\SysWOW64\ntdll.dll:??)
#2 0x75100a91 WaitForSingleObjectEx() (C:\Windows\syswow64\KernelBase.dll:??)
#3 0x3a0 ?? () (??:??)
#4 ?? ?? () (??:??)
Thread is stuck in a WaitForSingleObject (a standard windows multitasking primivitive, sort of select with only one handle)
Stack seems wired. But it's probably GDB that is wired and is lost between kernel/user call (or Windows that is not as GDB expect as you prefear, don't mind). Trying to force backtrace to start from specific address end in a GDB crash ... so as deadlock didn't occurs enought often, I do not follow this way

With ProcessExplorer, I catch the whole frame :
Quote
Code: [Select]
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x6c0
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x4a8
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64KiUserCallbackDispatcher+0x204
wow64win.dll+0x3798c
ntdll.dll!KiUserCallbackDispatcher+0x1f
wow64win.dll+0x3fe3a
wow64win.dll+0x1aea8
wow64.dll!Wow64SystemServiceEx+0xd7
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x2d
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64LdrpInitialize+0x429
ntdll.dll!RtlUniform+0x6e6
ntdll.dll!RtlCreateTagHeap+0xa7
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!ZwWaitForSingleObject+0x15
kernel32.dll!WaitForSingleObjectEx+0x43     <=======
kernel32.dll!WaitForSingleObject+0x12   <=======
wxmsw28u_gcc_cb.dll!_ZN17wxCriticalSection5LeaveEv+0x85
codecompletion.dll+0xd1c7 <=======
codecompletion.dll+0xd54f
codecompletion.dll+0x4a071
wxmsw28u_gcc_cb.dll!_ZNK12wxAppConsole11HandleEventEP12wxEvtHandlerMS0_FvR7wxEventES3_+0x22
codecompletion.dll+0x80a8d
codecompletion.dll+0x813ae
wxmsw28u_gcc_cb.dll!_ZNK12wxAppConsole11HandleEventEP12wxEvtHandlerMS0_FvR7wxEventES3_+0x22
wxmsw28u_gcc_cb.dll!_ZN12wxEvtHandler20ProcessPendingEventsEv+0xa1
wxmsw28u_gcc_cb.dll!_ZN12wxAppConsole20ProcessPendingEventsEv+0x40
ntdll.dll!KiUserCallbackDispatcher+0x2e
wxmsw28u_gcc_cb.dll!_ZN11wxEventLoop8DispatchEv+0x7c
wxmsw28u_gcc_cb.dll!_ZN17wxEventLoopManual3RunEv+0x128
wxmsw28u_gcc_cb.dll!_Z14wxUninitializev+0x181
codeblocks.exe+0x3898
codeblocks.exe+0x10fd
ntdll.dll!RtlInitializeExceptionChain+0x63
ntdll.dll!RtlInitializeExceptionChain+0x36
The interesting point is that at this stage is that it end at codecompletion.dll. Obviously ProcessExplorer use PDB symbols where GCC use DWARF. I have a tool to convert them, but it crash on such large file

Playing with dll offset and info symbol in GDB lead me to :
ClassBrowser::OnViewScope(wxCommandEvent&) + 599 in section .text of D:\bat\data\program\gcc\codeblocks\src\devel\share\codeblocks\plugins\codecompletion.dll
[Note that I'm not 100% sure of offset calculation]

Now, the other thread involved is (GDB trace) :
Quote
Code: [Select]
#0 0x75f272b9 USER32!GetPropW() (C:\Windows\syswow64\user32.dll:??)
#1 0x75f296c5 USER32!SendMessageW() (C:\Windows\syswow64\user32.dll:??)
#2 0x6cdbf2a7 wxTreeCtrl::GetItemParam(wxTreeItemId const&) const() (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#3 0xbe4fac8 ?? () (??:??)
#4 0x6cdbfe99 wxTreeCtrl::DoExpand(wxTreeItemId const&, int) () (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#5 0x6cdc0047 wxTreeCtrl::CollapseAndReset(wxTreeItemId const&) () (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#6 0xbe4fd60 ?? () (??:??)
#7 0x5828336 ClassBrowserBuilderThread::BuildTree(this=this@entry=0x7578c30) (D:\bat\data\program\gcc\codeblocks\src\plugins\codecompletion\classbrowserbuilderthread.cpp:550)
#8 0x582883d ClassBrowserBuilderThread::Entry(this=0x7578c30) (D:\bat\data\program\gcc\codeblocks\src\plugins\codecompletion\classbrowserbuilderthread.cpp:254)
#9 0x6ccd727c wxThreadInternal::DoThreadStart(wxThread*) () (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#10 0x6ccd731f wxThreadInternal::WinThreadStart(void*)@4() (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#11 0xbe4ff10 ?? () (??:??)
So it's ClassBrowserBuilderThread that is involved. It's currently refreshing the ClassBrowser wxTree, that end to a "SendMessage" in windows. SendMessage in a thread will wait that message pump that handle the specific windows process the message
But the message pump is in main thread ... and it's waiting something else, my guess is that it's waiting ClassBrowser to process something, ending in a classical deadlock.

At this point, it can be :
-a specific Windows issue (or not)
-a wx issue (lock/unlock) issue
-a usage issue

I've added 2 invasive trace

1)One in
Code: [Select]
bool Manager::ProcessEvent(CodeBlocksEvent& event)to track if issue is during a specific C::B event process.
Answer : No
Over 7 recorded deadlock it's nearly always after a call to JumpTracker plugin (so it's related to some input/editor move)
Code: [Select]
353188229     0     CALL 10364:M11JumpTrackerFvR15CodeBlocksEventE
353188229     0 END CALL 10364:M11JumpTrackerFvR15CodeBlocksEventE
One time it after a call to debugger plugin
Code: [Select]
1195030     0     CALL 10361:M16cbDebuggerPluginFvR15CodeBlocksEventE
 1195030     0 END CALL 10361:M16cbDebuggerPluginFvR15CodeBlocksEventE

2)One for logging all wx Event
Code: [Select]
int CodeBlocksApp::FilterEvent(wxEvent& event)I had to remove some of them occuring very often from log (or it end to hundred MB files in one minute)
Here, it always end up on an event "10183", that is "wxEVT_COMMAND_MENU_SELECTED" on my wx

Searching a bit tell me that this event seems to be send when something is clicked in a menu, but also as a mean to send inter thread event. In sdk_events.h
Code: [Select]
// Thread event, this is basically a derived wxCommandEvent but enforce a deep copy of its
// m_cmdString member. wxEVT_COMMAND_MENU_SELECTED is reused and event handlers are matched by
// ids. This is just to conserve the old code, an alternative is use some
// new event type like: cbEVT_THREAD_LOG_MESSAGE, cbEVT_THREAD_LOGDEBUG_MESSAGE
// cbEVT_THREAD_SYSTEM_HEADER_UPDATE.

For now I just catched the id associated with event one time :
Code: [Select]
EVT 10183 id:1463 ref:0x00000000
I will follow my way, but remarks and idea are welcomed to be more effective.

Online oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 10262
Re: Deadlock of C::B in CodeCompletion
« Reply #1 on: June 13, 2017, 02:24:18 pm »
Just disable the classbrowser in the settings of the code completion. The class browser thread is doing UI operations from a thread. This is dangerous and causes random problems. For wx28 it is a bit better, but in wx30 builds it is total disaster and thus the classbrowser is disabled there.

This code needs to be reimplemented from scratch, but there is no one that is doing the work at the moment. :(
<debugger plugin maintainer>
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 4968
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Deadlock of C::B in CodeCompletion
« Reply #2 on: June 13, 2017, 06:38:32 pm »
There are a lot of issues in CodeCompletion plugin, one issue is the symbol browser, which is running some gui functions from either GUI main thread or some worker threads. The other issue comes from the "lockers" which try to protect the "Parser" object or the "TokenTree" object, I think those lockers should be totally removed, but since I don't find a clean way to remove it.

The code is quite complex, and the only way maybe like OBF said, we should start from scratch.

If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Online oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 10262
Re: Deadlock of C::B in CodeCompletion
« Reply #3 on: June 13, 2017, 08:15:06 pm »
I'm talking about re-writing the class browser and not the whole CC plugin. We need an ClassBrowser API that is language agnostic and can work with more than one plugin. The current version is duplicated in the fortran plugin, which is rather ugly.
<debugger plugin maintainer>
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Bat

  • Multiple posting newcomer
  • *
  • Posts: 47
Re: Deadlock of C::B in CodeCompletion
« Reply #4 on: June 13, 2017, 08:51:50 pm »
I've currently disabled the ClassBrowser, to do a "clean test". So let's wait for next hang (or not)

For ClassBrowser from what I see, to not have lock, all UI operation should be pushed in main UI thread. But it will need to be separated in very unitary operation (remove a node, add a node) to not hurt UI performance, sot lot of overhead code to handle this

Online oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 10262
Re: Deadlock of C::B in CodeCompletion
« Reply #5 on: June 13, 2017, 11:12:43 pm »
In fact it should be done in batchs - update X nodes in a single event, then let some of the other events be processed, then update X more nodes and so on. The other option is switching to a virtual treectrl, but these are only available in wx3.x...
<debugger plugin maintainer>
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline yvesdm3000

  • Almost regular
  • **
  • Posts: 226
Re: Deadlock of C::B in CodeCompletion
« Reply #6 on: June 15, 2017, 06:05:42 pm »
In fact it should be done in batchs - update X nodes in a single event, then let some of the other events be processed, then update X more nodes and so on. The other option is switching to a virtual treectrl, but these are only available in wx3.x...
One could keep current implementation for wx2.8 and switch to a virtual treectrl when on wx3.x ?

I would greatly welcome a language-agnostic class browser for ClangCC !

Yves
« Last Edit: June 15, 2017, 06:07:50 pm by yvesdm3000 »
Clang based code completion for Code::Blocks:   http://github.com/yvesdm3000/ClangLib

Offline Bat

  • Multiple posting newcomer
  • *
  • Posts: 47
Re: Deadlock of C::B in CodeCompletion
« Reply #7 on: June 15, 2017, 07:06:55 pm »
I used virtual treectrl (out of wx, directly on win32 code) long time ago. It's clear that performance is really good - but there are messages flowing asking for data. Theses messages must be answered without too much wait (else it will freeze UI), so a kind of double buffering must be put in place.
P.S. No crash since I disabled ClassBrowser

Online oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 10262
Re: Deadlock of C::B in CodeCompletion
« Reply #8 on: June 15, 2017, 07:49:22 pm »
One could keep current implementation for wx2.8 and switch to a virtual treectrl when on wx3.x ?
We don't have the man power to support/fix one implementation. You're talking about two. :)
<debugger plugin maintainer>
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline yvesdm3000

  • Almost regular
  • **
  • Posts: 226
Re: Deadlock of C::B in CodeCompletion
« Reply #9 on: June 16, 2017, 05:55:53 am »
One could keep current implementation for wx2.8 and switch to a virtual treectrl when on wx3.x ?
We don't have the man power to support/fix one implementation. You're talking about two. :)
Current implementation works fine on 2.8 (or at least seems to be), and for 3.0 we need a new one anyway... How long are we going to keep 2.8 alive ? How many are using 2.8 specific for the actual working classbrowser that could move to 3.0 once classbrowser has been reimplemented ?

Yves
Clang based code completion for Code::Blocks:   http://github.com/yvesdm3000/ClangLib

Online oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 10262
Re: Deadlock of C::B in CodeCompletion
« Reply #10 on: June 16, 2017, 08:23:40 am »
yvesdm3000: Please re-read the topic! Bat is using wx2.8 and he is having problems that are resolved when the class browser is disabled.

wx3.x is super buggy, but some people are forced to use it on linux. We don't force windows users to use wx3.x builds, so we don't know how buggy it is on windows.
<debugger plugin maintainer>
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]