Author Topic: changes between r7459 and r7460 slow down project load  (Read 8911 times)

Offline cbnewbie

  • Multiple posting newcomer
  • *
  • Posts: 25
changes between r7459 and r7460 slow down project load
« on: October 28, 2011, 09:12:28 am »
After an svn upgrade from version r7450 to r7520 some time ago I discovered that loading my projects takes much more time now. Usually I was impressed that my projects could be loaded within 2 seconds. But now loading the same project takes at least 30 seconds! Depending on the size of the projects I may have to wait several minutes...
After installing several svn versions I found out that for versions until r7459 the time for loading project files is amazingly fast, but from version r4760 on there is a huge delay in loading them. What did change in codeblocks and how can I disable this feature?

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5211
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: changes between r7459 and r7460 slow down project load
« Reply #1 on: October 28, 2011, 09:20:38 am »
After installing several svn versions I found out that for versions until r7459 the time for loading project files is amazingly fast, but from version r4760 on there is a huge delay in loading them. What did change in codeblocks and how can I disable this feature?
Did you mean: r7460?

But I can't find any reason why r7460 will be slow than 7459.

What's changed in r7460 looks like:
Code: [Select]
Index: trunk/src/sdk/cbproject.cpp
===================================================================
--- trunk/src/sdk/cbproject.cpp (revision 7459)
+++ trunk/src/sdk/cbproject.cpp (revision 7460)
@@ -72,7 +72,7 @@
 
     virtual wxDirTraverseResult OnDir(const wxString& dirname)
     {
-        return wxDIR_IGNORE;
+        return wxDIR_CONTINUE;
     }
 
     bool IsValidPath() { return m_IsValidPath; }
@@ -380,7 +380,7 @@
             if (dir.IsOpened())
             {
                 ProjectDirTraverser traverser;
-                dir.Traverse(traverser, wxEmptyString, wxDIR_FILES);
+                dir.Traverse(traverser, wxEmptyString, wxDIR_DIRS | wxDIR_FILES);
                 if (traverser.IsValidPath())
                     base = tmpbaseF;
             }

If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline cbnewbie

  • Multiple posting newcomer
  • *
  • Posts: 25
Re: changes between r7459 and r7460 slow down project load
« Reply #2 on: October 28, 2011, 10:27:38 am »
Thanks a lot! I simply changed those lines in r7520 back. Now loading my projects is quick as before (<2 sec). I have no idea why cb behaves like this but I'm glad I know how to overcome this problem in future svn releases.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 11589
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #3 on: October 28, 2011, 10:41:30 am »
I can confirm that loading large projects is very slow.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline jens

  • Administrator
  • Lives here!
  • *****
  • Posts: 7265
    • Jens' unofficial debian-repository for the Code::Blocks - IDE
Re: changes between r7459 and r7460 slow down project load
« Reply #4 on: October 28, 2011, 11:27:49 am »
I can confirm that loading large projects is very slow.
Not here, but it might depend onthe place where your project file is located.

The ProjectDirTraverser is only called from inside CalculateTopLevelPath, and that should not happen so often.

Can you apply the following patch and start C::B with "-d" parameter, and have alook ath the "Code::Blocks Debug".
You should see how often the traverser is called and how long it needed.

You can also try what happens if the CodeCompletion-plugin is disabled.

Code: [Select]
diff --git a/src/sdk/cbproject.cpp b/src/sdk/cbproject.cpp
index c50a046..ec7d528 100644
--- a/src/sdk/cbproject.cpp
+++ b/src/sdk/cbproject.cpp
@@ -348,6 +348,7 @@ void cbProject::CalculateCommonTopLevelPath()
     wxFileName base = GetBasePath() + sep;
     Manager::Get()->GetLogManager()->DebugLog(_T("Project's base path: ") + base.GetFullPath());
 
+    int traverser_count = 0;
     // this loop takes ~30ms for 1000 project files
     // it's as fast as it can get, considered that it used to take ~1200ms ;)
     // don't even bother making it faster - you can't :)
@@ -381,10 +382,13 @@ void cbProject::CalculateCommonTopLevelPath()
             wxDir dir(tmpbaseF.GetPath());
             if (dir.IsOpened())
             {
+                Manager::Get()->GetLogManager()->DebugLogError(F(_("We call ProjectDirTraverser %d times"), ++traverser_count));
+                wxStopWatch sw;
                 ProjectDirTraverser traverser;
                 dir.Traverse(traverser, wxEmptyString, wxDIR_DIRS | wxDIR_FILES);
                 if (traverser.IsValidPath())
                     base = tmpbaseF;
+                Manager::Get()->GetLogManager()->DebugLogError(F(_("traversing took %d ms"), (int)sw.Time()));
             }
         }
     }

Offline cbnewbie

  • Multiple posting newcomer
  • *
  • Posts: 25
Re: changes between r7459 and r7460 slow down project load
« Reply #5 on: October 28, 2011, 01:15:32 pm »
I attached parts of a log file where just the number of loaded files per project and your ProjectDirTraverser output is listed. As you can see traversing is terribly slow...

I am using manually patched "svn build rev 7520 (2011-10-22 10:29:36) gcc 4.4.3 Linux/unicode - 64 bit"

Offline jens

  • Administrator
  • Lives here!
  • *****
  • Posts: 7265
    • Jens' unofficial debian-repository for the Code::Blocks - IDE
Re: changes between r7459 and r7460 slow down project load
« Reply #6 on: October 28, 2011, 02:02:47 pm »
I attached parts of a log file where just the number of loaded files per project and your ProjectDirTraverser output is listed. As you can see traversing is terribly slow...

I am using manually patched "svn build rev 7520 (2011-10-22 10:29:36) gcc 4.4.3 Linux/unicode - 64 bit"

Can you provide a test project, where it is so slow.

I also added a counter for files and dirs touched by the traverser and this is my output for the linux kernel 2.6.35 with a project-file deep in the source tree (just for testing):
Quote
33115 files loaded
Done loading project in 4460ms
Project's base path: /home/jens/kernel-tmp.2.6.35/linux-source-2.6.35/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/test-kernel-deep/
We call ProjectDirTraverser 1 times
ProjectDirTraverse searched in 1998 Dirs and found 33117 files
traversing took 539 ms

CodeCompletion is disabled and I use a patch to enhance load-speed, but it does not affect the traverser, just the project loading time decreases a lot.

Offline jens

  • Administrator
  • Lives here!
  • *****
  • Posts: 7265
    • Jens' unofficial debian-repository for the Code::Blocks - IDE
Re: changes between r7459 and r7460 slow down project load
« Reply #7 on: October 28, 2011, 02:07:13 pm »
Here is the patch I used:
Code: [Select]
diff --git a/src/sdk/cbproject.cpp b/src/sdk/cbproject.cpp
index fbf109a..dafb4cc 100644
--- a/src/sdk/cbproject.cpp
+++ b/src/sdk/cbproject.cpp
@@ -58,11 +58,12 @@ namespace compatibility { typedef TernaryCondTypedef<wxMinimumVersion<2,5>::eval
 class ProjectDirTraverser : public wxDirTraverser
 {
 public:
-    ProjectDirTraverser() : m_IsValidPath(false)
+    ProjectDirTraverser() : m_IsValidPath(false) , countFiles(0), countDirs(0)
     {}
 
     virtual wxDirTraverseResult OnFile(const wxString& filename)
     {
+        ++countFiles;
         const FileType ft = FileTypeOf(filename);
         if (ft == ftCodeBlocksProject || ft == ftCodeBlocksWorkspace)
         {
@@ -74,13 +75,20 @@ public:
 
     virtual wxDirTraverseResult OnDir(const wxString& dirname)
     {
+        ++countDirs;
         return wxDIR_CONTINUE;
     }
 
-    bool IsValidPath() { return m_IsValidPath; }
+    bool IsValidPath()
+    {
+        Manager::Get()->GetLogManager()->DebugLogError(F(_("ProjectDirTraverse searched in %d Dirs and found %d files"), countDirs, countFiles));
+        return m_IsValidPath;
+    }
 
 private:
     bool m_IsValidPath;
+    int countFiles;
+    int countDirs;
 };
 
 // class constructor
@@ -351,6 +359,7 @@ void cbProject::CalculateCommonTopLevelPath()
     wxFileName base = GetBasePath() + sep;
     Manager::Get()->GetLogManager()->DebugLog(_T("Project's base path: ") + base.GetFullPath());
 
+    int traverser_count = 0;
     // this loop takes ~30ms for 1000 project files
     // it's as fast as it can get, considered that it used to take ~1200ms ;)
     // don't even bother making it faster - you can't :)
@@ -385,10 +394,13 @@ void cbProject::CalculateCommonTopLevelPath()
             wxDir dir(tmpbaseF.GetPath());
             if (dir.IsOpened())
             {
+                Manager::Get()->GetLogManager()->DebugLogError(F(_("We call ProjectDirTraverser %d times"), ++traverser_count));
+                wxStopWatch sw;
                 ProjectDirTraverser traverser;
                 dir.Traverse(traverser, wxEmptyString, wxDIR_DIRS | wxDIR_FILES);
                 if (traverser.IsValidPath())
                     base = tmpbaseF;
+                Manager::Get()->GetLogManager()->DebugLogError(F(_("traversing took %d ms"), (int)sw.Time()));
             }
         }
     }

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 11589
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #8 on: October 28, 2011, 02:44:38 pm »
Code: [Select]
...Once...
We call ProjectDirTraverser 1 times
traversing took 578 ms
.... Many more...
We call ProjectDirTraverser 1 times
traversing took 146 ms
This is what I get, but this is the second time I load the workspace.

Jens: Do you know a way I can revoke the caches? This is on linux.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 11589
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #9 on: October 28, 2011, 02:49:04 pm »
Code: [Select]
... once for the first projects in the workspace...
We call ProjectDirTraverser 1 times
traversing took 20608 ms
.... Many more...
We call ProjectDirTraverser 1 times
traversing took 146 ms
This is the same workspace, after I've dropped the caches with
Code: [Select]
sync && echo 3 > /proc/sys/vm/drop_caches
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline cbnewbie

  • Multiple posting newcomer
  • *
  • Posts: 25
Re: changes between r7459 and r7460 slow down project load
« Reply #10 on: October 28, 2011, 03:07:18 pm »
I discovered something interesting. For every new project in my workspace a "Project's common toplevel path" is found - which is the same for all of my projects within that workspace. So, the number of searched directories and files (2673/59591) is also the same for all projects. But it takes a lot of time to scan them every time.

One problem seems to be that all projects within my workspace refer to external source code outside the separate project trees.

Workspace: /home/prg/Workspace/local/work/codeblocks/
Projects: /home/prg/Workspace/local/work/codeblocks/prg/*
External source: /home/prg/Workspace/local/source/*

That's why the "Project's common toplevel path" is /home/prg/Workspace/local for all projects.

OK, I do not understand why this changes between r7459 and r7460 have been made. But if they remain, couldn't the search results be cached somehow for the common toplevel path?

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 11589
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #11 on: October 28, 2011, 03:15:55 pm »
There was an user who have reported a bug. See this topic for more details: http://forums.codeblocks.org/index.php/topic,15023.msg101286.html#msg101286
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline jens

  • Administrator
  • Lives here!
  • *****
  • Posts: 7265
    • Jens' unofficial debian-repository for the Code::Blocks - IDE
Re: changes between r7459 and r7460 slow down project load
« Reply #12 on: October 28, 2011, 05:17:03 pm »
I see the problem, but how should caching work ?
Caching is normally done by the OS.

I will see if I can implement the same functionality without the dir-traverser, as it was before revision 7307.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 11589
    • Travis build status
Re: changes between r7459 and r7460 slow down project load
« Reply #13 on: October 28, 2011, 06:12:38 pm »
cbnewbie: is it slow the second time you open your workspace?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline jens

  • Administrator
  • Lives here!
  • *****
  • Posts: 7265
    • Jens' unofficial debian-repository for the Code::Blocks - IDE
Re: changes between r7459 and r7460 slow down project load
« Reply #14 on: October 29, 2011, 10:45:54 am »
I started a new discussion about the ProjectDirTraverser in general:
http://forums.codeblocks.org/index.php/topic,15430.msg103592.html#msg103592

It contains a patch to be tried.

Offline cbnewbie

  • Multiple posting newcomer
  • *
  • Posts: 25
Re: changes between r7459 and r7460 slow down project load
« Reply #15 on: October 31, 2011, 08:30:14 am »
@ oBFusCATed
As you can see in my previously attached file cb_log.txt there are two things happening:

1. Loading the first project in the workspace takes a lot of time while the other projects are loaded faster.
2. The overall performance of loading is slow.

But I am lucky that all projects have the same common top level path (see previously attached cb_log_2.txt). That is why the remaining projects are loaded faster. A really bad situation happens if you have projects with different top level paths and - even worse - if at least one project has the top level path "/". The last case will result in a whole system scan/traverse every time you open this project.


@ all:
Is it really necessary to scan all directories below the common top level path? Wouldn't it be enough to scan all directories below the actual workspace path? I do not miss any functionality if I simply undo the changes from version r4760 where traversing via directories was introduced.

If traversing the top level directory is really necessary for some functionality of c::b which I do not use (and therefore don't miss) can't there be an project option to traverse either the workspace directory or the top level directory?

Is it at all necessary to traverse the top level directory every time you open a project?


@ jens:
I will try your new patch today.

Offline jens

  • Administrator
  • Lives here!
  • *****
  • Posts: 7265
    • Jens' unofficial debian-repository for the Code::Blocks - IDE
Re: changes between r7459 and r7460 slow down project load
« Reply #16 on: October 31, 2011, 09:11:40 am »
@ all:
Is it really necessary to scan all directories below the common top level path? Wouldn't it be enough to scan all directories below the actual workspace path? I do not miss any functionality if I simply undo the changes from version r4760 where traversing via directories was introduced.
I have introduced it, because projects where the project-file is somewhere in the folder-structure of the projects where broken.
I did not introduce the ProjectDirTraverser, I just fixed this issue with it.
See
If traversing the top level directory is really necessary for some functionality of c::b which I do not use (and therefore don't miss) can't there be an project option to traverse either the workspace directory or the top level directory?

Is it at all necessary to traverse the top level directory every time you open a project?
Calculating the toplevel path has to be done every time a project is opened, or a file is added/deleted, because the project might be changed from other user, from C::B or even manually or it might have moved on the drive or to a different drive.
Projects can be shared by many people via revision control system, like C::B's projects.
And there are surely more cases, where a recalculating is needed.

But if I am correct the traverser is not needed in any of these cases.

See the http://forums.codeblocks.org/index.php/topic,15430.msg103592.html#msg103592.

Please make further discussion about calculating the common toplevel path there.

Direct answers to questions in this thread should stay here I think.