Author Topic: Bottleneck on loading large amount of files (especially on linux)  (Read 6688 times)

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
See this thread for the the exact problem.

libfab has a workspace that opens about 280 files at start.

With trunk on linux it took about 2:56 minutes, on windows (virtual w2k-machine) it took only 0:32.
The wxAui-test-branch needs "only" 1:10 min, it has improvements to the utf-8 conversion and some general tweaks when loading files, especially large files.
The utf-8 changes are already discussed here: http://forums.codeblocks.org/index.php/topic,10159.0.html.

I did some more research and found another cause for the slow performance (I still don't know, why it's os much slower on windows, than on linux, but that's another thing).
In editcolourset.cpp in function Apply we apply the default style for a given language to all styles up to wxSCI_STYLE_MAX (255).

I moved SetStyleBits, so we can get the amount of stylebits needed by the actual lexer, and therefore the maximum amount of really used styles (default is 32).
That speeds up the loading time for the test-workspace for about 30 seconds (it needs about 0:40 min now), so it comes near the windows loading time.

The patch below is not the patch I posted in the other thread (it was a little late, when I worked on it last night).
I think it should not break any lexer, because it determines the amount of used styles after SetStyleBits.

It would be really nice, if other devs (and of course all our users) can have alook at this patch, because together with the other tweaks in the wxAui-test branch fileloading on linux is much, much faster than with trunk.
But I'm not a lexer-guru, so I am not 100% sure, if it can not break anything.

Code
Index: codeblocks.aui/src/sdk/editorcolourset.cpp
===================================================================
--- codeblocks.aui/src/sdk/editorcolourset.cpp    (Revision 5593)
+++ codeblocks.aui/src/sdk/editorcolourset.cpp    (Arbeitskopie)
@@ -430,11 +430,15 @@
     if (lang == HL_NONE)
         return;
 
-    // first load the default colours to all styles (ignoring some built-in styles)
+    // first load the default colours to all styles used by the actual lexer (ignoring some built-in styles)
     OptionColour* defaults = GetOptionByName(lang, _("Default"));
+    OptionSet& mset = m_Sets[lang];
+    control->SetLexer(mset.m_Lexers);
+    control->SetStyleBits(control->GetStyleBitsNeeded());
     if (defaults)
     {
-        for (int i = 0; i < wxSCI_STYLE_MAX; ++i)
+        int countStyles = 1 << control->GetStyleBits();
+        for (int i = 0; i < countStyles; ++i)
         {
             if (i < 33 || i > 39)
                 DoApplyStyle(control, i, defaults);
@@ -445,7 +449,6 @@
     // this makes sure it stays the correct colour
     control->StyleSetForeground(wxSCI_STYLE_LINENUMBER, wxSystemSettings::GetColour(wxSYS_COLOUR_BTNTEXT));
 
-    OptionSet& mset = m_Sets[lang];
     for (unsigned int i = 0; i < mset.m_Colours.GetCount(); ++i)
     {
         OptionColour* opt = mset.m_Colours.Item(i);
@@ -486,8 +489,6 @@
 //            }
         }
     }
-    control->SetLexer(mset.m_Lexers);
-    control->SetStyleBits(control->GetStyleBitsNeeded());
     for (int i = 0; i <= wxSCI_KEYWORDSET_MAX; ++i)
     {
         control->SetKeyWords(i, mset.m_Keywords[i]);

Any suggestions and arguments against or for this patch are welcome.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: Bottleneck on loading large amount of files (especially on linux)
« Reply #1 on: May 14, 2009, 01:37:32 pm »
Any suggestions and arguments against or for this patch are welcome.
I would love to do that hence the patch fails with:

patching file "src/sdk/editorcolourset.cpp"
Hunk #1 FAILED at 430.
Hunk #2 FAILED at 449.
Hunk #3 FAILED at 489.
3 out of 3 hunks FAILED -- saving rejects to file "src/sdk/editorcolourset.cpp.rej"

..on the wxAUI branch (as well as on trunk).
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: Bottleneck on loading large amount of files (especially on linux)
« Reply #2 on: May 14, 2009, 01:56:03 pm »
Any suggestions and arguments against or for this patch are welcome.
I would love to do that hence the patch fails with:

patching file "src/sdk/editorcolourset.cpp"
Hunk #1 FAILED at 430.
Hunk #2 FAILED at 449.
Hunk #3 FAILED at 489.
3 out of 3 hunks FAILED -- saving rejects to file "src/sdk/editorcolourset.cpp.rej"

..on the wxAUI branch (as well as on trunk).

Possibly a line-ending problem.
I attach a patch file created with svn diff.

If that still does not work, you have to do it manually, it's not really much.

[attachment deleted by admin]

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: Bottleneck on loading large amount of files (especially on linux)
« Reply #3 on: May 14, 2009, 02:33:09 pm »
Possibly a line-ending problem.
Yupp - that was it. New patch works well...

Uh - oh... line-endings. How many ages will we still suffer from them?! :?
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline libfab

  • Multiple posting newcomer
  • *
  • Posts: 45
Re: Bottleneck on loading large amount of files (especially on linux)
« Reply #4 on: May 14, 2009, 10:36:37 pm »
Hi Jens,
I applied your new patch by hand and got the workspace opening in 45s, same config parameters.
The workspace was compiled OK.
Fab

Offline libfab

  • Multiple posting newcomer
  • *
  • Posts: 45
Re: Bottleneck on loading large amount of files (especially on linux)
« Reply #5 on: May 15, 2009, 08:25:19 pm »
There's not much to gain from simplifying EditorColourSet::Apply or ::DoApplyStyle, or from inlining. I tested a few simplifications there (even brute-force preprocessing by macros) and the delta was slight. So further improvements should be coming from other sections of the code.
Fab