Author Topic: question about running parser in thread  (Read 13228 times)

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
question about running parser in thread
« on: January 03, 2014, 04:05:05 pm »
I'm not sure the reason.

I just did some test, if I run the batch parsing in the thread pool, then the whole C::B gui freeze. I can see CPU usage is 50%(My cpu has two cores) lasts for several seconds until all the parser is done.

So, the question is: if a worker thread is doing a heavy job(e.g. scanning/parsing files), will the main GUI thread get hang? I don't see the GUI related operation in the worker thread. Maybe, in this case, another working process is better than working thread?

Thanks.

If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: question about running parser in thread
« Reply #1 on: January 03, 2014, 04:16:19 pm »
Maybe, in this case, another working process is better than working thread?
I always suspect its the call of the compiler to get #defines and stuff like that.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: question about running parser in thread
« Reply #2 on: January 04, 2014, 07:10:34 am »
Maybe, in this case, another working process is better than working thread?
I always suspect its the call of the compiler to get #defines and stuff like that.

I don't think this is the reason. Calling the compiler plugin to get the macro definition and compiler default search paths are quick, and I see GUI still hangs *after* the macro and include paths collection stage.

I did some further testing, and found that the hang only happens when I have some cbEditor opened, while in the same time, the batch parsing is running in the worker thread. If I close all the editor, then reparse the project again or load the project again, I don't see GUI hangs while parsing.

Can you confirm this?

Any hints about the reason?

BTW: When testing, I also *disabled* the editor related event in the CodeCompletion source code, such as:
So, I try to avoid the lockers (lockers in parser or tokentree) in the GUI, also comment out toolbar operation and symbol tree building code.
Code

//    pm->RegisterEventSink(cbEVT_EDITOR_SAVE,          new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorSaveOrModified));
//    pm->RegisterEventSink(cbEVT_EDITOR_MODIFIED,      new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorSaveOrModified));
//    pm->RegisterEventSink(cbEVT_EDITOR_OPEN,          new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorOpen));
//    pm->RegisterEventSink(cbEVT_EDITOR_ACTIVATED,     new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorActivated));
//    pm->RegisterEventSink(cbEVT_EDITOR_TOOLTIP,       new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorTooltip));
//    pm->RegisterEventSink(cbEVT_EDITOR_CLOSE,         new cbEventFunctor<CodeCompletion, CodeBlocksEvent>(this, &CodeCompletion::OnEditorClosed));


So, my guess is that scintilla control use some kind of idle handling? When worker thread is running, those idle handler has no chance to execute?
« Last Edit: January 04, 2014, 07:16:57 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: question about running parser in thread
« Reply #3 on: January 04, 2014, 07:39:28 am »
.....
So, my guess is that scintilla control use some kind of idle handling? When worker thread is running, those idle handler has no chance to execute?
I think I catch the reason.
I just comment out those likes:

Code
--- a/src/plugins/codecompletion/codecompletion.cpp
+++ b/src/plugins/codecompletion/codecompletion.cpp
@@ -489,7 +489,7 @@ int idAutocompSelectTimer       = wxNewId();
 #define EDITOR_ACTIVATED_DELAY    300
 
 BEGIN_EVENT_TABLE(CodeCompletion, cbCodeCompletionPlugin)
-    EVT_UPDATE_UI_RANGE(idMenuCodeComplete, idCurrentProjectReparse, CodeCompletion::OnUpdateUI)
+//    EVT_UPDATE_UI_RANGE(idMenuCodeComplete, idCurrentProjectReparse, CodeCompletion::OnUpdateUI)
 
     EVT_MENU(idMenuCodeComplete,                   CodeCompletion::OnCodeComplete             )
     EVT_MENU(idMenuShowCallTip,                    CodeCompletion::OnShowCallTip              )
@@ -638,7 +638,7 @@ void CodeCompletion::OnAttach()
 
     // hook to editors
     EditorHooks::HookFunctorBase* myhook = new EditorHooks::HookFunctor<CodeCompletion>(this, &CodeCompletion::EditorEventHook);
-    m_EditorHookId = EditorHooks::RegisterHook(myhook);
+//    m_EditorHookId = EditorHooks::RegisterHook(myhook);

Then I don't see the GUI cbEditor hangs now.

Well, look at the source code:

Code
void CodeCompletion::OnUpdateUI(wxUpdateUIEvent& event)
{
    wxString NameUnderCursor;
    bool IsInclude = false;
    const bool HasNameUnderCursor = CodeCompletionHelper::EditorHasNameUnderCursor(NameUnderCursor, IsInclude);

    const bool HasEd = Manager::Get()->GetEditorManager()->GetBuiltinActiveEditor() != 0;
    if (m_EditMenu)
    {
        m_EditMenu->Enable(idMenuCodeComplete, HasEd);
        m_EditMenu->Enable(idMenuShowCallTip, HasEd);
        const bool RenameEnable = HasNameUnderCursor && !IsInclude && m_NativeParser.GetParser().Done();
        m_EditMenu->Enable(idMenuRenameSymbols, RenameEnable);
    }

    if (m_SearchMenu)
    {
        m_SearchMenu->Enable(idMenuGotoFunction,       HasEd);
        m_SearchMenu->Enable(idMenuGotoPrevFunction,   HasEd);
        m_SearchMenu->Enable(idMenuGotoNextFunction,   HasEd);

        const bool GotoEnable = HasNameUnderCursor && !IsInclude;
        m_SearchMenu->Enable(idMenuGotoDeclaration,    GotoEnable);
        m_SearchMenu->Enable(idMenuGotoImplementation, GotoEnable);
        const bool FindEnable = HasNameUnderCursor && !IsInclude && m_NativeParser.GetParser().Done();
        m_SearchMenu->Enable(idMenuFindReferences, FindEnable);
        const bool IncludeEnable = HasNameUnderCursor && IsInclude;
        m_SearchMenu->Enable(idMenuOpenIncludeFile, IncludeEnable);
    }
....

You see in the function: m_NativeParser.GetParser().Done(), there are dirty lockers
Code
bool Parser::Done()
{
    CC_LOCKER_TRACK_P_MTX_LOCK(ParserCommon::s_ParserMutex)

    bool done =    m_PriorityHeaders.empty()
                && m_SystemPriorityHeaders.empty()
                && m_BatchParseFiles.empty()
                && m_PredefinedMacros.IsEmpty()
                && !m_NeedMarkFileAsLocal
                && m_PoolTask.empty()
                && m_Pool.Done();

    CC_LOCKER_TRACK_P_MTX_UNLOCK(ParserCommon::s_ParserMutex)

    return done;
}

I'm not quite sure, but if the batch parsing is running, the s_ParserMutex should already locked, thus when you GUI code go to this, you get GUI locked.  ???

EDIT:
The GUI hang problem may happens when I apply the patch cc_includes_parsing.patch from this link Several improvements to Code Completion plugin




« Last Edit: January 04, 2014, 10:33:35 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: question about running parser in thread
« Reply #4 on: January 05, 2014, 10:54:53 am »
You see in the function: m_NativeParser.GetParser().Done(), there are dirty lockers
What happens, if you remove those lockers? They may not even be needed (assuming atomic functions for .empty() etc...)
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: question about running parser in thread
« Reply #5 on: January 05, 2014, 04:13:58 pm »
You see in the function: m_NativeParser.GetParser().Done(), there are dirty lockers
What happens, if you remove those lockers? They may not even be needed (assuming atomic functions for .empty() etc...)
std::list<wxString>::empty() is atomic? I don't think so.

Only cbThreadPool::Done() is atomic as I can see.

Code
inline bool cbThreadPool::Done() const
{
  wxMutexLocker lock(m_Mutex);
  return m_workingThreads == 0;
}

Anyway, the GUI hang issue mostly happens by a local patch of Huki cc_includes_parsing.patch.

Would you mind to show some comments on how we can handle those kind of lockers? (I hate those lockers :), one reason is that wxString in wx2.8 is too bad for a multiply thread application, second reason is those locker code make the source a bit hard to read and understand, third reason is they have some issue to cause GUI hang.......)


About the issue you said:
Maybe, in this case, another working process is better than working thread?
I always suspect its the call of the compiler to get #defines and stuff like that.
To solve this, we can put this step to a worker thread or a worker thread task(Our Parserthread class is not a thread, but a task witch can executed in the thread pool). This should avoid the GUI hang when batch parsing started, because it is currently done in the event handler(wxTimer handler) of the GUI.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.