Author Topic: CC: Expensive calls on opening new files  (Read 23859 times)

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
CC: Expensive calls on opening new files
« on: June 21, 2007, 11:12:05 pm »
When opening a new file, the following gets called:

Code
void CodeCompletion::OnEditorActivated(CodeBlocksEvent& event)
{
    EditorBase* eb = event.GetEditor();
    if (IsAttached() && m_InitDone)
    {
        m_NativeParsers.OnEditorActivated(eb);
        m_FunctionsParsingTimer.Start(1000, wxTIMER_ONE_SHOT); // one second delay should be ok
    }

    event.Skip();
}

which eventually results in this call

Code
bool Parser::ParseBufferForFunctions(const wxString& buffer)
{
    ParserThreadOptions opts;
    opts.wantPreprocessor = m_Options.wantPreprocessor;
    opts.useBuffer = true;
    opts.bufferSkipBlocks = true;
    opts.handleFunctions = true;
    ParserThread* thread = new ParserThread(this,
                                            buffer,
                                            false,
                                            opts,
                                            m_pTempTokens);
    return thread->Parse();
}

on the main thread??

this is really time consuming for big files...


also note this hook gets called hundreds if not thousands of times when opening a file (should shut down the event propagation in the sdk until the text has been inserted?):

Code
void CodeCompletion::EditorEventHook(cbEditor* editor, wxScintillaEvent& event)
{
    ConfigManager* cfg = Manager::Get()->GetConfigManager(_T("code_completion")); //EXPENSIVE??

    if (!IsAttached() ||
        !m_InitDone)// ||
        !cfg->ReadBool(_T("/use_code_completion"), true))  //cfg-Read... EXPENSIVE??
    {
        event.Skip();
        return;
    }
...

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #1 on: June 22, 2007, 01:30:16 am »
I patched cbeditor to restrict editorhook calls during file opens. CC reports only ~ 200 events with patch (vs ~700,000 unpatched) when loading a 9mb file. patch: http://developer.berlios.de/patch/index.php?func=detailpatch&patch_id=2072&group_id=5358

this roughly halves the time until you see the editor spring up (at which time function parsing begins on the main thread - i haven't attempted to fix this...)

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #2 on: June 22, 2007, 02:03:03 am »
PS: please move if this doesn't belong here (the thread stuff is more relevant to this sub-forum, perhaps)

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #3 on: June 22, 2007, 02:23:44 am »
dmoore: Please share more! I need to know everything.

OK, first of all: The "m_FunctionsParsingTimer.Start(1000, wxTIMER_ONE_SHOT);" was added so Code Completion would wait at least some time before starting to parse. If a file was added to the project, the timer would be reset and the countdown would start from -1000 ms again. When all files had finished opening, the timer would advance to 0 and parsing would begin. So yes, basically it was a way to delay the parsing until the project had been finished.

However, I didn't know anything about the hooks. What tests exactly did you do to realize this was called thousands of times?

Update: I added a debug log to cbEditor::OnScintillaEvent(wxScintillaEvent& event). It displays a message along with a cumulative number. The number i got was 17242 effective calls during project opening. You're right.

But I got a better solution than your patch. It's called ProjectManager::IsBusy(). Returns true when the project is loading/closing.
« Last Edit: June 22, 2007, 02:55:41 am by rickg22 »

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #4 on: June 22, 2007, 02:49:29 am »
What do you know, it works! :) It only gets 17 effective calls during project opening.

I'll do something similar to code compleition's OnEditorActivated. Hats off for your discovery. :)

Update: Guess what, biplab had already fixed that part. I'm just cleaning it up.
« Last Edit: June 22, 2007, 03:21:47 am by rickg22 »

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #5 on: June 22, 2007, 03:35:50 am »
yes, please clean up as you see fit (would it be possible to switch off the event responder altogether the save the function calls?)

unfortunately, I can't say I used any special tool to find the problem other than C::B itself (lots of find in files and find implementation of). I know that on Linux you can use callgrind (part of valgrind) to produce call statistics, but I'm not familiar enough with that tool or the C::B code to use it effectively (you have to have an idea of what code should be called thousands of times and what shouldn't).

The main reason I was able find the problem came from playing around with big files and from recent stuff I learned while playing with scintilla (adding the php lexer and my wild goose chase with wxIdleEvents). Scintilla is very event driven, so it was just a case of figuring out which events were creating lags or conflicts... there may still be more issues that callgrind might be able to unearth.

It also helps that you guys have writen very clear modular code! :) (despite wxwidgets macro heavy implementation)

back to the Function Parsing code - delaying till all project files are open is good, but then it appears that the parsing occurs on the main thread, which blocks the UI until it's done, which kind of defeats the purpose of waiting...

and next problem: after the editor loses then regains focus, the function parser appears to reparse the whole file if you click on the class/function browser drop down. should only happen if file is modified right? (unless the data is released when the file loses focus?)

Offline Biplab

  • Developer
  • Lives here!
  • *****
  • Posts: 1874
    • Biplab's Blog
Re: CC: Expensive calls on opening new files
« Reply #6 on: June 22, 2007, 05:08:35 am »
unfortunately, I can't say I used any special tool to find the problem other than C::B itself (lots of find in files and find implementation of). I know that on Linux you can use callgrind (part of valgrind) to produce call statistics, but I'm not familiar enough with that tool or the C::B code to use it effectively (you have to have an idea of what code should be called thousands of times and what shouldn't).

I'll post a Callgrind log tonight (Now it's morning for me), too, if I can set it up properly. :)

back to the Function Parsing code - delaying till all project files are open is good, but then it appears that the parsing occurs on the main thread, which blocks the UI until it's done, which kind of defeats the purpose of waiting...

and next problem: after the editor loses then regains focus, the function parser appears to reparse the whole file if you click on the class/function browser drop down. should only happen if file is modified right? (unless the data is released when the file loses focus?)

These problems needs attention. :)
Be a part of the solution, not a part of the problem.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #7 on: June 22, 2007, 06:50:28 am »

back to the Function Parsing code - delaying till all project files are open is good, but then it appears that the parsing occurs on the main thread, which blocks the UI until it's done, which kind of defeats the purpose of waiting...

Actually the parsing isn't that slow. It's just one file. What makes it so slow is the event handling related to it. But if you wish, we can [ EDIT: at a *LATER* stage*, hope that clears it up, Don Corleone ;-) ] fix that, too. Volunteers? (I have thread-phobia :P )

and next problem: after the editor loses then regains focus, the function parser appears to reparse the whole file if you click on the class/function browser drop down. should only happen if file is modified right? (unless the data is released when the file loses focus?)

Yes, I noticed that. Curiously, this was the same redundant behavior that I *just* fixed in TODO list. So I'm changing it so it will parse *ALL* the open files at once, and just switch from the results whenever you change the active file.
« Last Edit: June 22, 2007, 03:38:15 pm by rickg22 »

Offline mandrav

  • Project Leader
  • Administrator
  • Lives here!
  • *****
  • Posts: 4315
    • Code::Blocks IDE
Re: CC: Expensive calls on opening new files
« Reply #8 on: June 22, 2007, 10:11:01 am »
I must admit I 'm kind of scared with some of the things I read here...
I 'm not saying you don't have a point: you may very well be right.

BUT...

I 'm afraid of such optimizations at such a late stage by people who are not very familiar with the particular code. And don't take this as an insult please. You all remember the time when CC was responsible for every bad thing in C::B. We don't want to go back there, do we?

All I 'm saying is that if you want to "play" around with the code and try to optimize things you 're very welcome to do so. But in a separate branch of the code. I don't want the risk of ruining what we already have, even if it's not performing optimally.

So, if it's a new branch you need created, just ask me and I will create one for you.
Thanks again for your efforts :).
Be patient!
This bug will be fixed soon...

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #9 on: June 22, 2007, 03:22:26 pm »
Yiannis: Don't worry about the thread problems! I'm not touching THAT thing (eew :-P ). It was just a suggestion, I have to admit sometimes I get misinterpreted (ah, remember the old times? :) ). I said "we" as "the team" because I don't want to meddle with multithreading. I still have open sores about that  ;-)

However, about prefetching the function lists, it's a simple change, I already tried in the todo list, and it worked wonders (that was BEFORE i had discovered the huge memory leak).

Rest assured, I'm not letting the multithreaded creature go rampant again  :lol:
« Last Edit: June 22, 2007, 03:30:29 pm by rickg22 »

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #10 on: June 22, 2007, 03:46:03 pm »
rick: let me know when you have finished making your planned changes (if you make them). after that I might release the multithreaded beast again in my local copy  :lol:

Offline David Perfors

  • Developer
  • Lives here!
  • *****
  • Posts: 560
Re: CC: Expensive calls on opening new files
« Reply #11 on: June 22, 2007, 03:51:22 pm »
Yiannis: Don't worry about the thread problems! I'm not touching THAT thing (eew :-P ). It was just a suggestion, I have to admit sometimes I get misinterpreted (ah, remember the old times? :) ). I said "we" as "the team" because I don't want to meddle with multithreading. I still have open sores about that  ;-)
Yes I remember the old times :lol: but it seems that you have learned a little bit of it :) (How do I know? you have just edited you're own text :P)
OS: winXP
Compiler: mingw
IDE: Code::Blocks SVN WX: 2.8.4 Wish list: faster code completion, easier debugging, refactoring

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #12 on: June 24, 2007, 02:32:11 am »
[
back to the Function Parsing code - delaying till all project files are open is good, but then it appears that the parsing occurs on the main thread, which blocks the UI until it's done, which kind of defeats the purpose of waiting...

and next problem: after the editor loses then regains focus, the function parser appears to reparse the whole file if you click on the class/function browser drop down. should only happen if file is modified right? (unless the data is released when the file loses focus?)

These problems needs attention. :)

Good news everybody, I patched that part. Now you can switch files as fast as you want, and the files won't get re-parsed.
It's in revision 4163.

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #13 on: June 24, 2007, 03:19:09 am »
good work rick.  :D

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #14 on: June 24, 2007, 03:29:38 am »
But I got a better solution than your patch. It's called ProjectManager::IsBusy(). Returns true when the project is loading/closing.

what about opening a file when no project is loaded? I assume that projectmanager won't be IsBusy() during that time? (could be wrong about this)

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #15 on: June 24, 2007, 04:49:37 am »
IsBusy only is set when ProjectManager is loading or closing a project. So if you don't have open projects, it'll be fine, too :)

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #16 on: June 24, 2007, 01:58:15 pm »
then won't that mean all those scintilla events get processed when opening a big file outside of a project?

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #17 on: June 24, 2007, 04:55:52 pm »
Opening a file, even big, should be no problem, because the code completion events are only triggered ONCE per file per save. That's my fix.

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #18 on: June 24, 2007, 05:07:25 pm »
won't EditorEventHook be called irrespective of whether parsing is actually taking place? (it's registered within the scope of the entire plugin, not at the individual parser level)

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #19 on: June 24, 2007, 05:10:19 pm »
additionally other plugins that have an EditorEventHook will still get called during large file loads outside of projects. When using CB as a general purpose editor/viewer, short load times are very desirable.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #20 on: June 25, 2007, 08:05:59 am »
If you use Code::Blocks as an editor, then you can disable the plugins. Codeblocks has an option, /profile=<profile_name>. You can specify profiles for usage and disable the plugins in those profiles. But I don't remember where to set that up... :(


Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #21 on: June 25, 2007, 12:28:10 pm »
yes, I use personalities frequently -- an excellent feature. I'm not pushing this point to aggravate you: it just seems like a simple fix to block those editor event hook calls during file opens as well as project opens...

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #22 on: June 25, 2007, 04:09:42 pm »
how long is your file in question? Can I have a copy for later tests?

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: CC: Expensive calls on opening new files
« Reply #23 on: June 26, 2007, 08:36:54 pm »
any large (4-5mb) syntax highlightable file will do (e.g. html, xml or , if you can find one, cpp file). I'm pretty sure editorhooks aren't registered on a per editor basis that discriminates by filetype. In fact it might be better to test on a non-cpp file to distinguish cycles spent on real parsing from redundant cycles spent in the editor hooks. for what it's worth, with CC on the scintilla events probably don't add more than about 30% to opening times, but given the simplicity of the fix I think it is worthwhile.

it would be even better to use a buffered loading model (like SciTE) that progressively adds to the file content, so that files appear instantaneously (integration with plugins would be tricky)

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: CC: Expensive calls on opening new files
« Reply #24 on: June 26, 2007, 09:08:21 pm »
So in other words you're editing a 4MB html file???? O.O

Ok, now I need to know. What exactly is the EXTENSION of the file you're editing? Also, are you trying to tell me that the editorhooks are called 4 million of times, one per character?
« Last Edit: June 26, 2007, 09:12:22 pm by rickg22 »