Deadlock of C::B in CodeCompletion

[Bat]

My configuration : custom C::B build (quite some modification), 32 bits, Win7, wx2.8.12.
I can modify/rebuild wx and C::B

I randomly (*) end in a deadlock, C::B unresponsive, not eating any processor
(*)can be 5 in a row, just after opening C::B or last for one week to appear. Seems to be related to activity - the more agressive I am on keyboard, the more it seems to occurs. I've no easy reproducable steps.
I would nearly say that it has nearly always occured, it's possible I see it more often now.

So to debug this, I launch C::B in GDB and work with that [quite a bit slow, on codecompletion specifically]
I share my findings :

So I just deadlockeded(*), I break process in GDB :
(*) It's not my first deadlock trace, and all are consistent with following trace

main thread stack trace

Code

#0 0x7790f8b1	ntdll!ZwWaitForSingleObject() (C:\Windows\SysWOW64\ntdll.dll:??)
#1 0x7790f8b1	ntdll!ZwWaitForSingleObject() (C:\Windows\SysWOW64\ntdll.dll:??)
#2 0x75100a91	WaitForSingleObjectEx() (C:\Windows\syswow64\KernelBase.dll:??)
#3 0x3a0	?? () (??:??)
#4 ??	?? () (??:??)

Thread is stuck in a WaitForSingleObject (a standard windows multitasking primivitive, sort of select with only one handle)
Stack seems wired. But it's probably GDB that is wired and is lost between kernel/user call (or Windows that is not as GDB expect as you prefear, don't mind). Trying to force backtrace to start from specific address end in a GDB crash ... so as deadlock didn't occurs enought often, I do not follow this way

With ProcessExplorer, I catch the whole frame :

Code

wow64cpu.dll!TurboDispatchJumpAddressEnd+0x6c0
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x4a8
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64KiUserCallbackDispatcher+0x204
wow64win.dll+0x3798c
ntdll.dll!KiUserCallbackDispatcher+0x1f
wow64win.dll+0x3fe3a
wow64win.dll+0x1aea8
wow64.dll!Wow64SystemServiceEx+0xd7
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x2d
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64LdrpInitialize+0x429
ntdll.dll!RtlUniform+0x6e6
ntdll.dll!RtlCreateTagHeap+0xa7
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!ZwWaitForSingleObject+0x15
kernel32.dll!WaitForSingleObjectEx+0x43     <=======
kernel32.dll!WaitForSingleObject+0x12   <=======
wxmsw28u_gcc_cb.dll!_ZN17wxCriticalSection5LeaveEv+0x85
codecompletion.dll+0xd1c7 <=======
codecompletion.dll+0xd54f
codecompletion.dll+0x4a071
wxmsw28u_gcc_cb.dll!_ZNK12wxAppConsole11HandleEventEP12wxEvtHandlerMS0_FvR7wxEventES3_+0x22
codecompletion.dll+0x80a8d
codecompletion.dll+0x813ae
wxmsw28u_gcc_cb.dll!_ZNK12wxAppConsole11HandleEventEP12wxEvtHandlerMS0_FvR7wxEventES3_+0x22
wxmsw28u_gcc_cb.dll!_ZN12wxEvtHandler20ProcessPendingEventsEv+0xa1
wxmsw28u_gcc_cb.dll!_ZN12wxAppConsole20ProcessPendingEventsEv+0x40
ntdll.dll!KiUserCallbackDispatcher+0x2e
wxmsw28u_gcc_cb.dll!_ZN11wxEventLoop8DispatchEv+0x7c
wxmsw28u_gcc_cb.dll!_ZN17wxEventLoopManual3RunEv+0x128
wxmsw28u_gcc_cb.dll!_Z14wxUninitializev+0x181
codeblocks.exe+0x3898
codeblocks.exe+0x10fd
ntdll.dll!RtlInitializeExceptionChain+0x63
ntdll.dll!RtlInitializeExceptionChain+0x36

The interesting point is that at this stage is that it end at codecompletion.dll. Obviously ProcessExplorer use PDB symbols where GCC use DWARF. I have a tool to convert them, but it crash on such large file

Playing with dll offset and info symbol in GDB lead me to :
ClassBrowser::OnViewScope(wxCommandEvent&) + 599 in section .text of D:\bat\data\program\gcc\codeblocks\src\devel\share\codeblocks\plugins\codecompletion.dll
[Note that I'm not 100% sure of offset calculation]

Now, the other thread involved is (GDB trace) :

Code

#0 0x75f272b9	USER32!GetPropW() (C:\Windows\syswow64\user32.dll:??)
#1 0x75f296c5	USER32!SendMessageW() (C:\Windows\syswow64\user32.dll:??)
#2 0x6cdbf2a7	wxTreeCtrl::GetItemParam(wxTreeItemId const&) const() (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#3 0xbe4fac8	?? () (??:??)
#4 0x6cdbfe99	wxTreeCtrl::DoExpand(wxTreeItemId const&, int) () (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#5 0x6cdc0047	wxTreeCtrl::CollapseAndReset(wxTreeItemId const&) () (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#6 0xbe4fd60	?? () (??:??)
#7 0x5828336	ClassBrowserBuilderThread::BuildTree(this=this@entry=0x7578c30) (D:\bat\data\program\gcc\codeblocks\src\plugins\codecompletion\classbrowserbuilderthread.cpp:550)
#8 0x582883d	ClassBrowserBuilderThread::Entry(this=0x7578c30) (D:\bat\data\program\gcc\codeblocks\src\plugins\codecompletion\classbrowserbuilderthread.cpp:254)
#9 0x6ccd727c	wxThreadInternal::DoThreadStart(wxThread*) () (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#10 0x6ccd731f	wxThreadInternal::WinThreadStart(void*)@4() (D:\bat\data\program\gcc\codeblocks\src\devel\wxmsw28u_gcc_cb.dll:??)
#11 0xbe4ff10	?? () (??:??)

So it's ClassBrowserBuilderThread that is involved. It's currently refreshing the ClassBrowser wxTree, that end to a "SendMessage" in windows. SendMessage in a thread will wait that message pump that handle the specific windows process the message
But the message pump is in main thread ... and it's waiting something else, my guess is that it's waiting ClassBrowser to process something, ending in a classical deadlock.

At this point, it can be :
-a specific Windows issue (or not)
-a wx issue (lock/unlock) issue
-a usage issue

I've added 2 invasive trace

1)One in

Code

bool Manager::ProcessEvent(CodeBlocksEvent& event)

to track if issue is during a specific C::B event process.
Answer : No
Over 7 recorded deadlock it's nearly always after a call to JumpTracker plugin (so it's related to some input/editor move)

Code

353188229     0     CALL 10364:M11JumpTrackerFvR15CodeBlocksEventE
353188229     0 END CALL 10364:M11JumpTrackerFvR15CodeBlocksEventE

One time it after a call to debugger plugin

Code

 1195030     0     CALL 10361:M16cbDebuggerPluginFvR15CodeBlocksEventE
 1195030     0 END CALL 10361:M16cbDebuggerPluginFvR15CodeBlocksEventE

2)One for logging all wx Event

Code

int CodeBlocksApp::FilterEvent(wxEvent& event)

I had to remove some of them occuring very often from log (or it end to hundred MB files in one minute)
Here, it always end up on an event "10183", that is "wxEVT_COMMAND_MENU_SELECTED" on my wx

Searching a bit tell me that this event seems to be send when something is clicked in a menu, but also as a mean to send inter thread event. In sdk_events.h

Code

// Thread event, this is basically a derived wxCommandEvent but enforce a deep copy of its
// m_cmdString member. wxEVT_COMMAND_MENU_SELECTED is reused and event handlers are matched by
// ids. This is just to conserve the old code, an alternative is use some
// new event type like: cbEVT_THREAD_LOG_MESSAGE, cbEVT_THREAD_LOGDEBUG_MESSAGE
// cbEVT_THREAD_SYSTEM_HEADER_UPDATE.

For now I just catched the id associated with event one time :

Code

EVT 10183 id:1463 ref:0x00000000

I will follow my way, but remarks and idea are welcomed to be more effective.

[oBFusCATed]

Just disable the classbrowser in the settings of the code completion. The class browser thread is doing UI operations from a thread. This is dangerous and causes random problems. For wx28 it is a bit better, but in wx30 builds it is total disaster and thus the classbrowser is disabled there.

This code needs to be reimplemented from scratch, but there is no one that is doing the work at the moment.

[ollydbg]

There are a lot of issues in CodeCompletion plugin, one issue is the symbol browser, which is running some gui functions from either GUI main thread or some worker threads. The other issue comes from the "lockers" which try to protect the "Parser" object or the "TokenTree" object, I think those lockers should be totally removed, but since I don't find a clean way to remove it.

The code is quite complex, and the only way maybe like OBF said, we should start from scratch.

[oBFusCATed]

I'm talking about re-writing the class browser and not the whole CC plugin. We need an ClassBrowser API that is language agnostic and can work with more than one plugin. The current version is duplicated in the fortran plugin, which is rather ugly.

[Bat]

I've currently disabled the ClassBrowser, to do a "clean test". So let's wait for next hang (or not)

For ClassBrowser from what I see, to not have lock, all UI operation should be pushed in main UI thread. But it will need to be separated in very unitary operation (remove a node, add a node) to not hurt UI performance, sot lot of overhead code to handle this

[oBFusCATed]

In fact it should be done in batchs - update X nodes in a single event, then let some of the other events be processed, then update X more nodes and so on. The other option is switching to a virtual treectrl, but these are only available in wx3.x...

[yvesdm3000]

In fact it should be done in batchs - update X nodes in a single event, then let some of the other events be processed, then update X more nodes and so on. The other option is switching to a virtual treectrl, but these are only available in wx3.x...

One could keep current implementation for wx2.8 and switch to a virtual treectrl when on wx3.x ?

I would greatly welcome a language-agnostic class browser for ClangCC !

Yves

[Bat]

I used virtual treectrl (out of wx, directly on win32 code) long time ago. It's clear that performance is really good - but there are messages flowing asking for data. Theses messages must be answered without too much wait (else it will freeze UI), so a kind of double buffering must be put in place.
P.S. No crash since I disabled ClassBrowser

[oBFusCATed]

One could keep current implementation for wx2.8 and switch to a virtual treectrl when on wx3.x ?

We don't have the man power to support/fix one implementation. You're talking about two.

[yvesdm3000]

One could keep current implementation for wx2.8 and switch to a virtual treectrl when on wx3.x ?

We don't have the man power to support/fix one implementation. You're talking about two.

Current implementation works fine on 2.8 (or at least seems to be), and for 3.0 we need a new one anyway... How long are we going to keep 2.8 alive ? How many are using 2.8 specific for the actual working classbrowser that could move to 3.0 once classbrowser has been reimplemented ?

Yves

[oBFusCATed]

yvesdm3000: Please re-read the topic! Bat is using wx2.8 and he is having problems that are resolved when the class browser is disabled.

wx3.x is super buggy, but some people are forced to use it on linux. We don't force windows users to use wx3.x builds, so we don't know how buggy it is on windows.