Author Topic: changes between r7459 and r7460 slow down project load  (Read 13406 times)

Offline cbnewbie

  • Multiple posting newcomer
  • *
  • Posts: 25
changes between r7459 and r7460 slow down project load
« on: October 28, 2011, 09:12:28 am »
After an svn upgrade from version r7450 to r7520 some time ago I discovered that loading my projects takes much more time now. Usually I was impressed that my projects could be loaded within 2 seconds. But now loading the same project takes at least 30 seconds! Depending on the size of the projects I may have to wait several minutes...
After installing several svn versions I found out that for versions until r7459 the time for loading project files is amazingly fast, but from version r4760 on there is a huge delay in loading them. What did change in codeblocks and how can I disable this feature?

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: changes between r7459 and r7460 slow down project load
« Reply #1 on: October 28, 2011, 09:20:38 am »
After installing several svn versions I found out that for versions until r7459 the time for loading project files is amazingly fast, but from version r4760 on there is a huge delay in loading them. What did change in codeblocks and how can I disable this feature?
Did you mean: r7460?

But I can't find any reason why r7460 will be slow than 7459.

What's changed in r7460 looks like:
Code
Index: trunk/src/sdk/cbproject.cpp
===================================================================
--- trunk/src/sdk/cbproject.cpp (revision 7459)
+++ trunk/src/sdk/cbproject.cpp (revision 7460)
@@ -72,7 +72,7 @@
 
     virtual wxDirTraverseResult OnDir(const wxString& dirname)
     {
-        return wxDIR_IGNORE;
+        return wxDIR_CONTINUE;
     }
 
     bool IsValidPath() { return m_IsValidPath; }
@@ -380,7 +380,7 @@
             if (dir.IsOpened())
             {
                 ProjectDirTraverser traverser;
-                dir.Traverse(traverser, wxEmptyString, wxDIR_FILES);
+                dir.Traverse(traverser, wxEmptyString, wxDIR_DIRS | wxDIR_FILES);
                 if (traverser.IsValidPath())
                     base = tmpbaseF;
             }

If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline cbnewbie

  • Multiple posting newcomer
  • *
  • Posts: 25
Re: changes between r7459 and r7460 slow down project load
« Reply #2 on: October 28, 2011, 10:27:38 am »
Thanks a lot! I simply changed those lines in r7520 back. Now loading my projects is quick as before (<2 sec). I have no idea why cb behaves like this but I'm glad I know how to overcome this problem in future svn releases.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #3 on: October 28, 2011, 10:41:30 am »
I can confirm that loading large projects is very slow.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: changes between r7459 and r7460 slow down project load
« Reply #4 on: October 28, 2011, 11:27:49 am »
I can confirm that loading large projects is very slow.
Not here, but it might depend onthe place where your project file is located.

The ProjectDirTraverser is only called from inside CalculateTopLevelPath, and that should not happen so often.

Can you apply the following patch and start C::B with "-d" parameter, and have alook ath the "Code::Blocks Debug".
You should see how often the traverser is called and how long it needed.

You can also try what happens if the CodeCompletion-plugin is disabled.

Code
diff --git a/src/sdk/cbproject.cpp b/src/sdk/cbproject.cpp
index c50a046..ec7d528 100644
--- a/src/sdk/cbproject.cpp
+++ b/src/sdk/cbproject.cpp
@@ -348,6 +348,7 @@ void cbProject::CalculateCommonTopLevelPath()
     wxFileName base = GetBasePath() + sep;
     Manager::Get()->GetLogManager()->DebugLog(_T("Project's base path: ") + base.GetFullPath());
 
+    int traverser_count = 0;
     // this loop takes ~30ms for 1000 project files
     // it's as fast as it can get, considered that it used to take ~1200ms ;)
     // don't even bother making it faster - you can't :)
@@ -381,10 +382,13 @@ void cbProject::CalculateCommonTopLevelPath()
             wxDir dir(tmpbaseF.GetPath());
             if (dir.IsOpened())
             {
+                Manager::Get()->GetLogManager()->DebugLogError(F(_("We call ProjectDirTraverser %d times"), ++traverser_count));
+                wxStopWatch sw;
                 ProjectDirTraverser traverser;
                 dir.Traverse(traverser, wxEmptyString, wxDIR_DIRS | wxDIR_FILES);
                 if (traverser.IsValidPath())
                     base = tmpbaseF;
+                Manager::Get()->GetLogManager()->DebugLogError(F(_("traversing took %d ms"), (int)sw.Time()));
             }
         }
     }

Offline cbnewbie

  • Multiple posting newcomer
  • *
  • Posts: 25
Re: changes between r7459 and r7460 slow down project load
« Reply #5 on: October 28, 2011, 01:15:32 pm »
I attached parts of a log file where just the number of loaded files per project and your ProjectDirTraverser output is listed. As you can see traversing is terribly slow...

I am using manually patched "svn build rev 7520 (2011-10-22 10:29:36) gcc 4.4.3 Linux/unicode - 64 bit"

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: changes between r7459 and r7460 slow down project load
« Reply #6 on: October 28, 2011, 02:02:47 pm »
I attached parts of a log file where just the number of loaded files per project and your ProjectDirTraverser output is listed. As you can see traversing is terribly slow...

I am using manually patched "svn build rev 7520 (2011-10-22 10:29:36) gcc 4.4.3 Linux/unicode - 64 bit"

Can you provide a test project, where it is so slow.

I also added a counter for files and dirs touched by the traverser and this is my output for the linux kernel 2.6.35 with a project-file deep in the source tree (just for testing):
Quote
33115 files loaded
Done loading project in 4460ms
Project's base path: /home/jens/kernel-tmp.2.6.35/linux-source-2.6.35/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/test-kernel-deep/
We call ProjectDirTraverser 1 times
ProjectDirTraverse searched in 1998 Dirs and found 33117 files
traversing took 539 ms

CodeCompletion is disabled and I use a patch to enhance load-speed, but it does not affect the traverser, just the project loading time decreases a lot.

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: changes between r7459 and r7460 slow down project load
« Reply #7 on: October 28, 2011, 02:07:13 pm »
Here is the patch I used:
Code
diff --git a/src/sdk/cbproject.cpp b/src/sdk/cbproject.cpp
index fbf109a..dafb4cc 100644
--- a/src/sdk/cbproject.cpp
+++ b/src/sdk/cbproject.cpp
@@ -58,11 +58,12 @@ namespace compatibility { typedef TernaryCondTypedef<wxMinimumVersion<2,5>::eval
 class ProjectDirTraverser : public wxDirTraverser
 {
 public:
-    ProjectDirTraverser() : m_IsValidPath(false)
+    ProjectDirTraverser() : m_IsValidPath(false) , countFiles(0), countDirs(0)
     {}
 
     virtual wxDirTraverseResult OnFile(const wxString& filename)
     {
+        ++countFiles;
         const FileType ft = FileTypeOf(filename);
         if (ft == ftCodeBlocksProject || ft == ftCodeBlocksWorkspace)
         {
@@ -74,13 +75,20 @@ public:
 
     virtual wxDirTraverseResult OnDir(const wxString& dirname)
     {
+        ++countDirs;
         return wxDIR_CONTINUE;
     }
 
-    bool IsValidPath() { return m_IsValidPath; }
+    bool IsValidPath()
+    {
+        Manager::Get()->GetLogManager()->DebugLogError(F(_("ProjectDirTraverse searched in %d Dirs and found %d files"), countDirs, countFiles));
+        return m_IsValidPath;
+    }
 
 private:
     bool m_IsValidPath;
+    int countFiles;
+    int countDirs;
 };
 
 // class constructor
@@ -351,6 +359,7 @@ void cbProject::CalculateCommonTopLevelPath()
     wxFileName base = GetBasePath() + sep;
     Manager::Get()->GetLogManager()->DebugLog(_T("Project's base path: ") + base.GetFullPath());
 
+    int traverser_count = 0;
     // this loop takes ~30ms for 1000 project files
     // it's as fast as it can get, considered that it used to take ~1200ms ;)
     // don't even bother making it faster - you can't :)
@@ -385,10 +394,13 @@ void cbProject::CalculateCommonTopLevelPath()
             wxDir dir(tmpbaseF.GetPath());
             if (dir.IsOpened())
             {
+                Manager::Get()->GetLogManager()->DebugLogError(F(_("We call ProjectDirTraverser %d times"), ++traverser_count));
+                wxStopWatch sw;
                 ProjectDirTraverser traverser;
                 dir.Traverse(traverser, wxEmptyString, wxDIR_DIRS | wxDIR_FILES);
                 if (traverser.IsValidPath())
                     base = tmpbaseF;
+                Manager::Get()->GetLogManager()->DebugLogError(F(_("traversing took %d ms"), (int)sw.Time()));
             }
         }
     }

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #8 on: October 28, 2011, 02:44:38 pm »
Code
...Once...
We call ProjectDirTraverser 1 times
traversing took 578 ms
.... Many more...
We call ProjectDirTraverser 1 times
traversing took 146 ms
This is what I get, but this is the second time I load the workspace.

Jens: Do you know a way I can revoke the caches? This is on linux.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #9 on: October 28, 2011, 02:49:04 pm »
Code
... once for the first projects in the workspace...
We call ProjectDirTraverser 1 times
traversing took 20608 ms
.... Many more...
We call ProjectDirTraverser 1 times
traversing took 146 ms
This is the same workspace, after I've dropped the caches with
Code
sync && echo 3 > /proc/sys/vm/drop_caches
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline cbnewbie

  • Multiple posting newcomer
  • *
  • Posts: 25
Re: changes between r7459 and r7460 slow down project load
« Reply #10 on: October 28, 2011, 03:07:18 pm »
I discovered something interesting. For every new project in my workspace a "Project's common toplevel path" is found - which is the same for all of my projects within that workspace. So, the number of searched directories and files (2673/59591) is also the same for all projects. But it takes a lot of time to scan them every time.

One problem seems to be that all projects within my workspace refer to external source code outside the separate project trees.

Workspace: /home/prg/Workspace/local/work/codeblocks/
Projects: /home/prg/Workspace/local/work/codeblocks/prg/*
External source: /home/prg/Workspace/local/source/*

That's why the "Project's common toplevel path" is /home/prg/Workspace/local for all projects.

OK, I do not understand why this changes between r7459 and r7460 have been made. But if they remain, couldn't the search results be cached somehow for the common toplevel path?

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #11 on: October 28, 2011, 03:15:55 pm »
There was an user who have reported a bug. See this topic for more details: http://forums.codeblocks.org/index.php/topic,15023.msg101286.html#msg101286
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: changes between r7459 and r7460 slow down project load
« Reply #12 on: October 28, 2011, 05:17:03 pm »
I see the problem, but how should caching work ?
Caching is normally done by the OS.

I will see if I can implement the same functionality without the dir-traverser, as it was before revision 7307.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #13 on: October 28, 2011, 06:12:38 pm »
cbnewbie: is it slow the second time you open your workspace?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: changes between r7459 and r7460 slow down project load
« Reply #14 on: October 29, 2011, 10:45:54 am »
I started a new discussion about the ProjectDirTraverser in general:
http://forums.codeblocks.org/index.php/topic,15430.msg103592.html#msg103592

It contains a patch to be tried.