Author Topic: observation on memory consumption  (Read 65540 times)

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: observation on memory consumption
« Reply #30 on: January 27, 2006, 08:24:08 pm »
Code::Blocks loading CodeBlocks.cbp with code completion turned on: 85.6 MB

In my case it goes up to more than 120 MB (around 110000 tokens in a bit more than 1100 files). Before it was a bit more than 90 MB (a bit more than 90000 tokens in a bit more than 900 files). This increase was remarked since rickg22 commited some code completion fixes. To be noted is that I have all the options selected in code completion (cache included).

Michael

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #31 on: January 27, 2006, 08:27:59 pm »
In my case it goes up to more than 120 MB (around 110000 tokens in a bit more than 1100 files).
That's 14.68 MB in tokens versus 108 MB in "something else", so it is even worse.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: observation on memory consumption
« Reply #32 on: January 27, 2006, 08:39:18 pm »
In my case it goes up to more than 120 MB (around 110000 tokens in a bit more than 1100 files).
That's 14.68 MB in tokens versus 108 MB in "something else", so it is even worse.

Yes. May be rickg22's fixes have caused code completion to read more files, which explains the higher number of tokens and of memory used. I am trying to continue the tests with Purify, but now I got Purify crashing when running C::B with just the option "memory leaks"... :?

Michael

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: observation on memory consumption
« Reply #33 on: January 27, 2006, 09:22:38 pm »
Well, besides the token i have some extra search structures, but the search tree only uses memory linear to the number of tokens added, and it's less than the token names.

Thomas, are you POSITIVELY sure about the space used per token? Because each token includes an amount of of strings, and the space used by the strings is not insignificant... I may, however, delete an extra search tree on the tokens' fullnames...

Yiannis: I haven't noticed crashes with codecompletion, please post stack traces - it's probably due to using more than one thread at the same time (before my fixes it only used one thread), so maybe there are critical sections missing. Maybe the reason for CB not crashing on my machine is that I have a single-core... :-/

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #34 on: January 27, 2006, 09:32:57 pm »
Quote
Thomas, are you POSITIVELY sure about the space used per token? Because each token includes an amount of of strings, and the space used by the strings is not insignificant... I may, however, delete an extra search tree on the tokens' fullnames...
The space used by strings is allocated separately, it is not part of the Token object, and the block allocator has no control over it. In fact, if the string buffer were part of the object, the block allocator would crash, as it can only handle objects of fixed size with no derived virtual descendants.

sizeof(Token) is positively 140 in revision 1887 (used to be 124 in revision 1421).
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #35 on: January 27, 2006, 09:43:48 pm »
Quote
Maybe the reason for CB not crashing on my machine is that I have a single-core... :-/
My AMD64 is single-core as well, but I saw several crashes this afternoon, too. I can't reproduce them now in 1888, but I get a constant freeze when changing the parser settings (turning off "follow includes").
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline mandrav

  • Project Leader
  • Administrator
  • Lives here!
  • *****
  • Posts: 4315
    • Code::Blocks IDE
Re: observation on memory consumption
« Reply #36 on: January 27, 2006, 09:49:47 pm »
Yiannis: I haven't noticed crashes with codecompletion, please post stack traces

As I said: C::B just dies. I don't get a crash dialog or anything. Like a stack overflow or something...
Be patient!
This bug will be fixed soon...

Offline tiwag

  • Developer
  • Lives here!
  • *****
  • Posts: 1196
  • sailing away ...
    • tiwag.cb
Re: observation on memory consumption
« Reply #37 on: January 27, 2006, 10:20:22 pm »
i did another test:

CodeBlocks - all plugins disabled except for
*Compiler, *Debugger and *MIME Handler

then i continuously opened and closed CodeBlocks.cbp (rev 1888)

1.) first time with startpage enabled,
2.) second time without startpage.
the results are (almost) the same.

from test 1.)
14x CodeBlocks.cbp opened/closed

from the last 13 times we get a memory rise of 8.4MB (from 11.2 to 19.6 MB)
mean value of mem-rise = 8.4MB/13 =~ 0.65MB per open/close of the project.


[attachment deleted by admin]

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: observation on memory consumption
« Reply #38 on: January 28, 2006, 12:37:28 am »
So there's a leak elsewhere? O.o

Ok, try it without debugger / MIME handler now.

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: observation on memory consumption
« Reply #39 on: January 28, 2006, 01:38:32 pm »
Hello,

I have remarked something about memory leaks. I have tried to open/close several times the C::B project with all the plugins and I have remarked that each time I lose around 2MB. For example if at beginning I have X=2MB, then after opening and closing C::B project I have X=4MB. If I repeat I will have X=6MB and so on. Then I have tried without plugins. I have switched off all the plugins in the 2menage plugins". I have observed the same behavior. X=2MB, then X=4MB, then X=5MB, then X=7MB and so on. Practically, the memory needed to open the C::B project is not released when I close the workspace.

Hope this could help :).

Michael

[EDIT] Sorry I have forgotten to say that I have used rev1889.
« Last Edit: January 28, 2006, 01:43:47 pm by Michael »

sethjackson

  • Guest
Re: observation on memory consumption
« Reply #40 on: January 28, 2006, 02:28:56 pm »
Hello,

I have remarked something about memory leaks. I have tried to open/close several times the C::B project with all the plugins and I have remarked that each time I lose around 2MB. For example if at beginning I have X=2MB, then after opening and closing C::B project I have X=4MB. If I repeat I will have X=6MB and so on. Then I have tried without plugins. I have switched off all the plugins in the 2menage plugins". I have observed the same behavior. X=2MB, then X=4MB, then X=5MB, then X=7MB and so on. Practically, the memory needed to open the C::B project is not released when I close the workspace.

Hope this could help :).

Michael

[EDIT] Sorry I have forgotten to say that I have used rev1889.


I thought thomas siad it was the start here page or something. Although the memory for project opening/closing could be a problem too.......

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #41 on: January 28, 2006, 02:48:15 pm »
I thought thomas siad it was the start here page or something.
Well, that does not mean there can't be others...

I believe that I found and fixed the leak regarding project files. I still have to test it a couple of times before committing though.
This leak is wxWidget's fault, in my opinion. DeleteNode() suggests that this deletes the node and the data, even more so as the documentation to RemoveNode() says "you have to free the object yourself". I replaced the DeleteNode() with the code sample from the wxList documentation (basically just deletes the node and the object), and it seems that the memory leak is gone.

I get this now when opening/closing the Code::Blocks project (compiler and debugger plugin active, all others disabled, start_here page disabled)



Code
Index: sdk/cbproject.cpp
===================================================================
--- sdk/cbproject.cpp (revision 1889)
+++ sdk/cbproject.cpp (working copy)
@@ -989,24 +989,18 @@
             return false;
 
  // now free the rest of the project files
-    int count = m_Files.GetCount();
     Manager::Get()->GetEditorManager()->HideNotebook();
     FilesList::Node* node = m_Files.GetFirst();
     while(node)
     {
         ProjectFile* f = node->GetData();
-        if (Manager::Get()->GetEditorManager()->Close(f->file.GetFullPath(),true))
-        {
-            FilesList::Node* oldNode = node;
-            node = node->GetNext();
-            m_Files.DeleteNode(oldNode);
-            --count;
-        }
-        else
-            node = node->GetNext();
+        Manager::Get()->GetEditorManager()->Close(f->file.GetFullPath(),true);
+        delete f;
+        delete node;
+        node = m_Files.GetFirst();
     }
     Manager::Get()->GetEditorManager()->ShowNotebook();
-    return count == 0;
+    return true;
 }
 
 bool cbProject::SaveAllFiles()
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #42 on: January 28, 2006, 03:23:32 pm »
Memory consumption now seems to be almost constant, possibly asymptotic (with code completion disabled).

With code completion enabled, the base memory still grows linearly, but the peak stays constant (that is probably an effect of the block allocator). There is also quite noticeable heap fragmentation (visible by the dull peaks and the lessened inclination of the curve after opening and closing a large project 15-20 times. I haven't had the time to test what happens when the base memory grows as high as the peak, have to go out now... :(
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: observation on memory consumption
« Reply #43 on: January 28, 2006, 09:23:04 pm »
Hello,

I have performed some tests with the rev1891 and I have remarked the good works of the devs&all concerning the memory leaks :D. Unfortunately, the behvior that I have observed before it is not fully gone away.

I have tried C::B without any plugins and opened/closed several times the C::B project. After closing the project, the memory is used is returned to the system, but not always and when its amount is generally less than what it was used. Sometimes the memory got accumulated and later returned to the system. This behavior was relatively chaotic and difficult to correctly describe it (sorry). Before it did not returned the memory used when opening a project (the amount was relatively constant, i.e., 1-2MB). Now, the amount of memory used to open the project is sometimes 100-200 kb, sometimes less.

IMHO, there is still something that keeps the memory and does not or not always return it to the OS.

Michael 

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #44 on: January 28, 2006, 09:43:07 pm »
Quote
IMHO, there is still something that keeps the memory and does not or not always return it to the OS.
Lol, at the risk of repeating myself... the block allocator never returns allocated memory to the OS, this is normal.

Only objects that are allocated via standard operator new are returned to the OS when they are deleted. That is for example the case for classes like wxFileName, or wxTextCtrl, but it is not true for cbEvent, Token, ProjectFile, ProjectBuildtarget, and maybe a couple of others.

Thus, it is correct that not all memory is freed, it must not be. The important point, however, is that the memory is reused, so if you open the same project 20 times (and even if you open/close different projects several times), the total amount of memory has to stay the same as long as your total memory requirement is not higher.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."