question about running parser in thread

[ollydbg]

I'm not sure the reason.

I just did some test, if I run the batch parsing in the thread pool, then the whole C::B gui freeze. I can see CPU usage is 50%(My cpu has two cores) lasts for several seconds until all the parser is done.

So, the question is: if a worker thread is doing a heavy job(e.g. scanning/parsing files), will the main GUI thread get hang? I don't see the GUI related operation in the worker thread. Maybe, in this case, another working process is better than working thread?

Thanks.

[MortenMacFly]

Maybe, in this case, another working process is better than working thread?

I always suspect its the call of the compiler to get #defines and stuff like that.

[ollydbg]

Maybe, in this case, another working process is better than working thread?

I always suspect its the call of the compiler to get #defines and stuff like that.

I don't think this is the reason. Calling the compiler plugin to get the macro definition and compiler default search paths are quick, and I see GUI still hangs *after* the macro and include paths collection stage.

I did some further testing, and found that the hang only happens when I have some cbEditor opened, while in the same time, the batch parsing is running in the worker thread. If I close all the editor, then reparse the project again or load the project again, I don't see GUI hangs while parsing.

Can you confirm this?

Any hints about the reason?

BTW: When testing, I also *disabled* the editor related event in the CodeCompletion source code, such as:
So, I try to avoid the lockers (lockers in parser or tokentree) in the GUI, also comment out toolbar operation and symbol tree building code.

Code

//    pm->RegisterEventSink(cbEVT_EDITOR_SAVE,          new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorSaveOrModified));
//    pm->RegisterEventSink(cbEVT_EDITOR_MODIFIED,      new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorSaveOrModified));
//    pm->RegisterEventSink(cbEVT_EDITOR_OPEN,          new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorOpen));
//    pm->RegisterEventSink(cbEVT_EDITOR_ACTIVATED,     new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorActivated));
//    pm->RegisterEventSink(cbEVT_EDITOR_TOOLTIP,       new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorTooltip));
//    pm->RegisterEventSink(cbEVT_EDITOR_CLOSE,         new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorClosed));

So, my guess is that scintilla control use some kind of idle handling? When worker thread is running, those idle handler has no chance to execute?

[ollydbg]

.....
So, my guess is that scintilla control use some kind of idle handling? When worker thread is running, those idle handler has no chance to execute?

I think I catch the reason.
I just comment out those likes:

Code

--- a/src/plugins/codecompletion/codecompletion.cpp
+++ b/src/plugins/codecompletion/codecompletion.cpp
@@ -489,7 +489,7 @@ int idAutocompSelectTimer       = wxNewId();
 #define EDITOR_ACTIVATED_DELAY    300
 
 BEGIN_EVENT_TABLE(CodeCompletion, cbCodeCompletionPlugin)
-    EVT_UPDATE_UI_RANGE(idMenuCodeComplete, idCurrentProjectReparse, CodeCompletion::OnUpdateUI)
+//    EVT_UPDATE_UI_RANGE(idMenuCodeComplete, idCurrentProjectReparse, CodeCompletion::OnUpdateUI)
 
     EVT_MENU(idMenuCodeComplete,                   CodeCompletion::OnCodeComplete             )
     EVT_MENU(idMenuShowCallTip,                    CodeCompletion::OnShowCallTip              )
@@ -638,7 +638,7 @@ void CodeCompletion::OnAttach()
 
     // hook to editors
     EditorHooks::HookFunctorBase* myhook = new EditorHooks::HookFunctor<CodeCompletion>(this, &CodeCompletion::EditorEventHook);
-    m_EditorHookId = EditorHooks::RegisterHook(myhook);
+//    m_EditorHookId = EditorHooks::RegisterHook(myhook);

Then I don't see the GUI cbEditor hangs now.

Well, look at the source code:

Code

void CodeCompletion::OnUpdateUI(wxUpdateUIEvent& event)
{
    wxString NameUnderCursor;
    bool IsInclude = false;
    const bool HasNameUnderCursor = CodeCompletionHelper::EditorHasNameUnderCursor(NameUnderCursor, IsInclude);

    const bool HasEd = Manager::Get()->GetEditorManager()->GetBuiltinActiveEditor() != 0;
    if (m_EditMenu)
    {
        m_EditMenu->Enable(idMenuCodeComplete, HasEd);
        m_EditMenu->Enable(idMenuShowCallTip, HasEd);
        const bool RenameEnable = HasNameUnderCursor && !IsInclude && m_NativeParser.GetParser().Done();
        m_EditMenu->Enable(idMenuRenameSymbols, RenameEnable);
    }

    if (m_SearchMenu)
    {
        m_SearchMenu->Enable(idMenuGotoFunction,       HasEd);
        m_SearchMenu->Enable(idMenuGotoPrevFunction,   HasEd);
        m_SearchMenu->Enable(idMenuGotoNextFunction,   HasEd);

        const bool GotoEnable = HasNameUnderCursor && !IsInclude;
        m_SearchMenu->Enable(idMenuGotoDeclaration,    GotoEnable);
        m_SearchMenu->Enable(idMenuGotoImplementation, GotoEnable);
        const bool FindEnable = HasNameUnderCursor && !IsInclude && m_NativeParser.GetParser().Done();
        m_SearchMenu->Enable(idMenuFindReferences, FindEnable);
        const bool IncludeEnable = HasNameUnderCursor && IsInclude;
        m_SearchMenu->Enable(idMenuOpenIncludeFile, IncludeEnable);
    }
....

You see in the function: m_NativeParser.GetParser().Done(), there are dirty lockers

Code

bool Parser::Done()
{
    CC_LOCKER_TRACK_P_MTX_LOCK(ParserCommon::s_ParserMutex)

    bool done =    m_PriorityHeaders.empty()
                && m_SystemPriorityHeaders.empty()
                && m_BatchParseFiles.empty()
                && m_PredefinedMacros.IsEmpty()
                && !m_NeedMarkFileAsLocal
                && m_PoolTask.empty()
                && m_Pool.Done();

    CC_LOCKER_TRACK_P_MTX_UNLOCK(ParserCommon::s_ParserMutex)

    return done;
}

I'm not quite sure, but if the batch parsing is running, the s_ParserMutex should already locked, thus when you GUI code go to this, you get GUI locked.  

EDIT:
The GUI hang problem may happens when I apply the patch cc_includes_parsing.patch from this link Several improvements to Code Completion plugin

[MortenMacFly]

You see in the function: m_NativeParser.GetParser().Done(), there are dirty lockers

What happens, if you remove those lockers? They may not even be needed (assuming atomic functions for .empty() etc...)

[ollydbg]

You see in the function: m_NativeParser.GetParser().Done(), there are dirty lockers

What happens, if you remove those lockers? They may not even be needed (assuming atomic functions for .empty() etc...)

std::list<wxString>::empty() is atomic? I don't think so.

Only cbThreadPool::Done() is atomic as I can see.

Code

inline bool cbThreadPool::Done() const
{
  wxMutexLocker lock(m_Mutex);
  return m_workingThreads == 0;
}

Anyway, the GUI hang issue mostly happens by a local patch of Huki cc_includes_parsing.patch.

Would you mind to show some comments on how we can handle those kind of lockers? (I hate those lockers , one reason is that wxString in wx2.8 is too bad for a multiply thread application, second reason is those locker code make the source a bit hard to read and understand, third reason is they have some issue to cause GUI hang.......)


About the issue you said:

Maybe, in this case, another working process is better than working thread?

I always suspect its the call of the compiler to get #defines and stuff like that.

To solve this, we can put this step to a worker thread or a worker thread task(Our Parserthread class is not a thread, but a task witch can executed in the thread pool). This should avoid the GUI hang when batch parsing started, because it is currently done in the event handler(wxTimer handler) of the GUI.