Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => CodeCompletion redesign => Topic started by: tiwag on January 26, 2006, 08:52:48 pm

Title: observation on memory consumption
Post by: tiwag on January 26, 2006, 08:52:48 pm
what i've done:

codecompletion plugin settings:
codecompletion: enabled,
parser: local includes and preprocessor definitions

i started C::B, memory settled at 13MB
loaded CodeBlocks.cbp project, memory settled at 36MB (peak is somewhat higher)
closed project, memory settled at 23MB, #note: + 10MB
loaded CodeBlocks.cbp project, memory settled at 37MB,
....

[edit] typos "memeory"

so we get a series of

13-36-23-37-24-38-26-40-27-42-28-43-44-32-47-33  [MB] memory consumption only by opening/closing the project

mean value (without first +10MB rise) :  delta-memory/project-load = (33 -23)MB / 7 = ~1.43MB leaked memory
and this is proportional to the amount of memory which is needed for all of the parsed symbols.

( hint: those with 2GB of ram, try it with local & global includes switched on ;-)  )

[attachment deleted by admin]
Title: Re: observation on memory consumption
Post by: thomas on January 26, 2006, 09:10:28 pm
Hmm... this measurement is inaccurate in one point.

Parser tokens are blockallocated objects, so that memory is never really freed. The application retains the memory block even if the objects are freed and reuses it when the next project is opened. So it appears to be leaking even if everything works correctly.

The same thing is true for ProjectFile objects, and ProjectBuildTargets, which are both block allocated, too.
Title: Re: observation on memory consumption
Post by: rickg22 on January 26, 2006, 09:12:21 pm
OK Tiwag, try removing the blockallocated stuff and see if there's a leak. Is the leak still there? What happens if we also remove the codecompletion plugin?
Title: Re: observation on memory consumption
Post by: thomas on January 26, 2006, 09:17:22 pm
No need to remove the block allocator. There *is* a leak for sure.

I only said that the measurement is not accurate because of the block allocator, as it will always retain some memory. However, that memory does not grow and grow and grow indefinitely if you keep closing and re-opening a project. It always stays the same.

You may want to enable debug mode in the block allocator if you are interested whether it's Tokens what is leaked...
Title: Re: observation on memory consumption
Post by: thomas on January 26, 2006, 09:58:43 pm
Here is a more detailled explanation of why the block allocator can confuse you.

These are two recordings of the very same actions using the very same program and the same projects, but in different order:

(http://img358.imageshack.us/img358/5893/memory2sb.png)

1. Code::Blocks starts
2. Open wxSmith.cbp
3. Close wxSmith.cbp (here you get the idea "hey, what's happening, memory is leaked?")
4. Open CodeBlocks.cbp
5. Close CodeBlocks.cbp (again memory seems to be leaked)
6. Open wxSmith.cbp again (now it seems to consume no memory at all!)

Let's try the reverse order:

A. Code::Blocks starts
B. Open CodeBlocks.cbp
C. Close CodeBlocks.cbp
D. Open wxSmith.cbp

Again, opening and closing CodeBlocks.cbp seems to massively leak memory, but wxSmith.cbp does not seem to consume any memory at all.

This phenomenon is caused by the fact that the block allocator retains the memory and reuses it for the newly allocated objects. :)
Title: Re: observation on memory consumption
Post by: rickg22 on January 26, 2006, 11:03:34 pm
280: Where are your patches?
Title: Re: observation on memory consumption
Post by: rickg22 on January 27, 2006, 12:15:53 am
I don't care, just show us which lines are at fault...
Title: Re: observation on memory consumption
Post by: thomas on January 27, 2006, 01:20:19 am
Rick, you will be delighted to hear that the memory leak has nothing to do with code completion :)
Code completion takes up so much RAM that the actual leaks (which are a lot smaller) are not immediately visible at all.

The most surprising observation when sampling the application's memory without code completion is that the amount of allocated memory actually rises by a good amount when you close a project!

I have been quite puzzled by that observation for a while, but I found out why: It is the start here page.
Apparently, a new page is allocated every time it is shown (and it is never freed). I'll look into the code tomorrow. Too tired to do that now, wouldn't want to break something... :)
Title: Re: observation on memory consumption
Post by: rickg22 on January 27, 2006, 01:32:04 am
Yay! Thanks! :)
Title: Re: observation on memory consumption
Post by: Game_Ender on January 27, 2006, 03:03:40 am
Do any of the three core developers run Codeblocks on linux routinely?  If they do I would suggested putting Codeblocks through valgrind (http://www.valgrind.org), I will try to do this tonight. My guess is it won't be so pretty.  One problem though is that wxWidgets itself has some small memory allocation errors, which happers the error finding processs. 
Title: Re: observation on memory consumption
Post by: rickg22 on January 27, 2006, 06:04:31 am
sorry, i don't... :(

by the way, i updated Codecompletion again today.
Title: Re: observation on memory consumption
Post by: Michael on January 27, 2006, 12:48:42 pm
Hello,

I have done some test with C::B (rev1884) about memory and remarked the following. At the start up, C::B memory requirement goes up to 27-28 MB. When I open the C::B project, C::B memory requirement goes up to 121-122 MB (with code completion). When closing the project (by using "close workspace"), the 50% of the memory used is returned. The same behavior happen without code completion, only that the used memory is much less.

IMHO there are some memory problems with both C::B itself and with code completion plugin.

Michael
Title: Re: observation on memory consumption
Post by: thomas on January 27, 2006, 01:16:01 pm
When closing the project (by using "close workspace"), the 50% of the memory used is returned. The same behavior happen without code completion, only that the used memory is much less.
This is normal behaviour. About 50% of the memory belongs to block allocated objects, these are never freed. Thus it always looks like there is a massive leak at first glance.

You cannot find out about memory leaks as simple as this (see my above post with the memory diagram). If you want to detect a memory leak by observing the total memory consumption, then you have to open/close the same project many times (10-20 times) as tiwag did.
If there are no memory leaks, you will still see that only 50% of the memory is freed, but if there is a leak (as it is the case), 50% of the peak will be freed every time, but the base will grow linearly.

Other options would be to
a) Compile wxWidgets in debug mode and run Code::Blocks from console. This will detect leaks of non-blockallocated objects.

b) Recompile Code::Blocks using the block allocator in debug mode. This will detect leaks of blockallocated objects. To do this, pass a positive integer as an additional template parameter to the classes which you want to observe:
  class Token  : public BlockAllocated<Token, 10000>
becomes
  class Token  : public BlockAllocated<Token, 10000, 1>

c) Use valgrind, but it is a lot easier to recompile wxWidgets in debug mode
Title: Re: observation on memory consumption
Post by: Michael on January 27, 2006, 02:07:03 pm
When closing the project (by using "close workspace"), the 50% of the memory used is returned. The same behavior happen without code completion, only that the used memory is much less.
This is normal behaviour. About 50% of the memory belongs to block allocated objects, these are never freed. Thus it always looks like there is a massive leak at first glance.

You cannot find out about memory leaks as simple as this (see my above post with the memory diagram). If you want to detect a memory leak by observing the total memory consumption, then you have to open/close the same project many times (10-20 times) as tiwag did.
If there are no memory leaks, you will still see that only 50% of the memory is freed, but if there is a leak (as it is the case), 50% of the peak will be freed every time, but the base will grow linearly.

Thank you for your answer :). I have not open/close the same project 10-20 times, but it is sometimes already that I check C::B memory consumption when opening/closing the C::B project. Interesting is that after closing it and do some work with another application (e.g., IE, Word) and later checking the memory consumption of C::B, the value is of only 2-4 MB.

Anyway, I have tried C::B with Purify. You can find the report(s) in this topic (http://forums.codeblocks.org/index.php?topic=2164.0).

Michael

Title: Re: observation on memory consumption
Post by: tiwag on January 27, 2006, 02:08:16 pm
Quote
... but it is a lot easier to recompile wxWidgets in debug mode

to build CB with wx-debug libs is easy, but it doesn't run,

i've done this already but couldn't solve the never ending assertions ...

it seems, that any eventhandler inside CB throws assertions all the time - if this could be solved, we can debug the memory leaks
Title: Re: observation on memory consumption
Post by: thomas on January 27, 2006, 02:16:59 pm
Quote
Interesting is that after closing it and do some work with another application (e.g., IE, Word) and later checking the memory consumption of C::B, the value is of only 2-4 MB.
This is a Windows malfunction. The Windows marketing FUD for this is "working set optimization" or something, but it is really a malfunction.

If you minimize an application, Windows purges most of its memory pages, no matter whether it needs the RAM or not. The "reasoning" behind this is that it is good to have a lot of free RAM, but in fact it only proves that they don't know the difference between "nano" and "milli". Purging memory pages when you don't need to is really, really stupid.
Title: Re: observation on memory consumption
Post by: mandrav on January 27, 2006, 02:22:11 pm
Quote
... but it is a lot easier to recompile wxWidgets in debug mode

to build CB with wx-debug libs is easy, but it doesn't run,

i've done this already but couldn't solve the never ending assertions ...

it seems, that any eventhandler inside CB throws assertions all the time - if this could be solved, we can debug the memory leaks

Strange, I have removed all the assertions I could find (last week, IIRC). Maybe it's been some time since you tried it?
Title: Re: observation on memory consumption
Post by: tiwag on January 27, 2006, 02:24:05 pm
Quote
... but it is a lot easier to recompile wxWidgets in debug mode

to build CB with wx-debug libs is easy, but it doesn't run,

i've done this already but couldn't solve the never ending assertions ...

it seems, that any eventhandler inside CB throws assertions all the time - if this could be solved, we can debug the memory leaks

Strange, I have removed all the assertions I could find (last week, IIRC). Maybe it's been some time since you tried it?

thx for the info,
last time i did was about 3 weeks before ;-)

 
Title: Re: observation on memory consumption
Post by: thomas on January 27, 2006, 03:43:56 pm
but it is a lot easier to recompile wxWidgets in debug mode
Well, I may have been a bit hasty in my judgement here. In fact, it is not easy to get it to work, at all. :lol:
Title: Re: observation on memory consumption
Post by: tiwag on January 27, 2006, 04:06:08 pm
i just finished with building a WXDEBUG build of C::B  (added __WXDEBUG__ to Project's Compiler #definitions, and changed custom variable WX_SUFFIX=ud )

it runs, but crashes when attempting to load CodeBlocks.cbp :(

but a small console project, which i have built with the NewProject generator, loads and works without problems ;-)
not all hope is lost ...
Title: Re: observation on memory consumption
Post by: rickg22 on January 27, 2006, 05:35:53 pm
Thomas, I have a request... are you sure you can't make a way to free VERY LARGE unused blocks of memory with your blockallocator? i.e. we have tokens allocated in groups of 10,000. Could you make it so that when all the tokens of a specific block have been freed, the block would be released automatically? This would spare us having 600Megs of unused memory making up space after we close the contribs workspace... ¬¬
Title: Re: observation on memory consumption
Post by: takeshimiya on January 27, 2006, 05:43:14 pm
Just to make sure I'm not doing anything wrong:
I open the contribs workspace with codecompletion activated. I don't do absolutely nothing, and the memory and cpu increases makes the system almost unuseable.

This is the graph.

[attachment deleted by admin]
Title: Re: observation on memory consumption
Post by: mandrav on January 27, 2006, 05:55:54 pm
Is this just me, or is code-completion crashy today?
Some times it will crash on project load, some other times it will crash on project close and other times (twice) has hung with 50% cpu usage indefinetely...

And when I say crash, I mean it just disappears. No error message (windows or otherwise), nothing.
Title: Re: observation on memory consumption
Post by: takeshimiya on January 27, 2006, 06:02:07 pm
Yes, codecompletion is very crashy, but he is not alone, debugger is also crashy :D (not that much like codecompletion, but...).

But codecompletion is really unuseable, it eats all the cpu, all the ram, and then, probably, crashes. :P


EDIT: when I mean that debugger is crashy, I mean, I only do a right-click on the watches panel (even without the debugger running), click on Add watch, and "it just disappears. No error message (windows or otherwise), nothing."
Title: Re: observation on memory consumption
Post by: mandrav on January 27, 2006, 06:13:39 pm
EDIT: when I mean that debugger is crashy, I mean, I only do a right-click on the watches panel (even without the debugger running), click on Add watch, and "it just disappears. No error message (windows or otherwise), nothing."

For this, you should update more often before complaining ;)
Title: Re: observation on memory consumption
Post by: tiwag on January 27, 2006, 06:37:32 pm
Is this just me, or is code-completion crashy today?...
the only crash i got today related to codecompletion was when i ran the wxdebug-version of CB ;-)
Title: Re: observation on memory consumption
Post by: takeshimiya on January 27, 2006, 07:15:52 pm
For this, you should update more often before complaining ;)
Mm, I'm at Rev.1887 which is the latest. That means that anonymous SVN is lagging?
Title: Re: observation on memory consumption
Post by: mandrav on January 27, 2006, 07:25:59 pm
That's strange because it really crashed earlier today but that was an error in the script bindings which I fixed. I have been debugging all day since that time...

Will look at it later again...
Title: Re: observation on memory consumption
Post by: takeshimiya on January 27, 2006, 07:32:06 pm
Just in case you can't reproduce it, I attach the backtrace.

[attachment deleted by admin]
Title: Re: observation on memory consumption
Post by: thomas on January 27, 2006, 08:06:59 pm
Thomas, I have a request... are you sure you can't make a way to free VERY LARGE unused blocks of memory with your blockallocator? i.e. we have tokens allocated in groups of 10,000. Could you make it so that when all the tokens of a specific block have been freed, the block would be released automatically? This would spare us having 600Megs of unused memory making up space after we close the contribs workspace... ¬¬
sizeof(Token) == 140, thus 600 MB == 4,491,264 Tokens. If you really allocate that many tokens, then you should think about looking for a serious bug in the parser ;)

But now seriously: we already talked about this idea of releasing blocks, and I already told you once: this is a really bad idea. Tokens, or their allocation is not the problem we're trying to solve. Starting to tweak the allocator will get you absolutely nowhere, it will only add to overhead.

Parsing CodeBlocks.cbp produces 79,000 tokens with "everything turned on". The block allocator allocates 80,000 tokens (10.68 MB) for these.

However, if you look at the actual memory consumption, you get figures like this:
Code::Blocks flat: 11.0 MB
Code::Blocks loading CodeBlocks.cbp with code completion turned off: 11.6 MB
Code::Blocks loading CodeBlocks.cbp with code completion turned on: 85.6 MB

Code completion uses up 74 MB, but only 10.68 MB are assigned to tokens. So where do the other 63.62 MB go? Do you see my point?

Adding the ability to release free blocks would make the allocator a lot more complicated and would mean a noticeable increase (read "noticeable " as "about 100%") in execution time for allocations and deallocations, and there is really no reason why we need it. The memory consumed by tokens is very acceptable compared to other structures. Also, the memory is not "lost", but reused when more objects are allocated.
Title: Re: observation on memory consumption
Post by: Michael on January 27, 2006, 08:24:08 pm
Code::Blocks loading CodeBlocks.cbp with code completion turned on: 85.6 MB

In my case it goes up to more than 120 MB (around 110000 tokens in a bit more than 1100 files). Before it was a bit more than 90 MB (a bit more than 90000 tokens in a bit more than 900 files). This increase was remarked since rickg22 commited some code completion fixes. To be noted is that I have all the options selected in code completion (cache included).

Michael
Title: Re: observation on memory consumption
Post by: thomas on January 27, 2006, 08:27:59 pm
In my case it goes up to more than 120 MB (around 110000 tokens in a bit more than 1100 files).
That's 14.68 MB in tokens versus 108 MB in "something else", so it is even worse.
Title: Re: observation on memory consumption
Post by: Michael on January 27, 2006, 08:39:18 pm
In my case it goes up to more than 120 MB (around 110000 tokens in a bit more than 1100 files).
That's 14.68 MB in tokens versus 108 MB in "something else", so it is even worse.

Yes. May be rickg22's fixes have caused code completion to read more files, which explains the higher number of tokens and of memory used. I am trying to continue the tests with Purify, but now I got Purify crashing when running C::B with just the option "memory leaks"... :?

Michael
Title: Re: observation on memory consumption
Post by: rickg22 on January 27, 2006, 09:22:38 pm
Well, besides the token i have some extra search structures, but the search tree only uses memory linear to the number of tokens added, and it's less than the token names.

Thomas, are you POSITIVELY sure about the space used per token? Because each token includes an amount of of strings, and the space used by the strings is not insignificant... I may, however, delete an extra search tree on the tokens' fullnames...

Yiannis: I haven't noticed crashes with codecompletion, please post stack traces - it's probably due to using more than one thread at the same time (before my fixes it only used one thread), so maybe there are critical sections missing. Maybe the reason for CB not crashing on my machine is that I have a single-core... :-/
Title: Re: observation on memory consumption
Post by: thomas on January 27, 2006, 09:32:57 pm
Quote
Thomas, are you POSITIVELY sure about the space used per token? Because each token includes an amount of of strings, and the space used by the strings is not insignificant... I may, however, delete an extra search tree on the tokens' fullnames...
The space used by strings is allocated separately, it is not part of the Token object, and the block allocator has no control over it. In fact, if the string buffer were part of the object, the block allocator would crash, as it can only handle objects of fixed size with no derived virtual descendants.

sizeof(Token) is positively 140 in revision 1887 (used to be 124 in revision 1421).
Title: Re: observation on memory consumption
Post by: thomas on January 27, 2006, 09:43:48 pm
Quote
Maybe the reason for CB not crashing on my machine is that I have a single-core... :-/
My AMD64 is single-core as well, but I saw several crashes this afternoon, too. I can't reproduce them now in 1888, but I get a constant freeze when changing the parser settings (turning off "follow includes").
Title: Re: observation on memory consumption
Post by: mandrav on January 27, 2006, 09:49:47 pm
Yiannis: I haven't noticed crashes with codecompletion, please post stack traces

As I said: C::B just dies. I don't get a crash dialog or anything. Like a stack overflow or something...
Title: Re: observation on memory consumption
Post by: tiwag on January 27, 2006, 10:20:22 pm
i did another test:

CodeBlocks - all plugins disabled except for
*Compiler, *Debugger and *MIME Handler

then i continuously opened and closed CodeBlocks.cbp (rev 1888)

1.) first time with startpage enabled,
2.) second time without startpage.
the results are (almost) the same.

from test 1.)
14x CodeBlocks.cbp opened/closed

from the last 13 times we get a memory rise of 8.4MB (from 11.2 to 19.6 MB)
mean value of mem-rise = 8.4MB/13 =~ 0.65MB per open/close of the project.


[attachment deleted by admin]
Title: Re: observation on memory consumption
Post by: rickg22 on January 28, 2006, 12:37:28 am
So there's a leak elsewhere? O.o

Ok, try it without debugger / MIME handler now.
Title: Re: observation on memory consumption
Post by: Michael on January 28, 2006, 01:38:32 pm
Hello,

I have remarked something about memory leaks. I have tried to open/close several times the C::B project with all the plugins and I have remarked that each time I lose around 2MB. For example if at beginning I have X=2MB, then after opening and closing C::B project I have X=4MB. If I repeat I will have X=6MB and so on. Then I have tried without plugins. I have switched off all the plugins in the 2menage plugins". I have observed the same behavior. X=2MB, then X=4MB, then X=5MB, then X=7MB and so on. Practically, the memory needed to open the C::B project is not released when I close the workspace.

Hope this could help :).

Michael

[EDIT] Sorry I have forgotten to say that I have used rev1889.
Title: Re: observation on memory consumption
Post by: sethjackson on January 28, 2006, 02:28:56 pm
Hello,

I have remarked something about memory leaks. I have tried to open/close several times the C::B project with all the plugins and I have remarked that each time I lose around 2MB. For example if at beginning I have X=2MB, then after opening and closing C::B project I have X=4MB. If I repeat I will have X=6MB and so on. Then I have tried without plugins. I have switched off all the plugins in the 2menage plugins". I have observed the same behavior. X=2MB, then X=4MB, then X=5MB, then X=7MB and so on. Practically, the memory needed to open the C::B project is not released when I close the workspace.

Hope this could help :).

Michael

[EDIT] Sorry I have forgotten to say that I have used rev1889.


I thought thomas siad it was the start here page or something. Although the memory for project opening/closing could be a problem too.......
Title: Re: observation on memory consumption
Post by: thomas on January 28, 2006, 02:48:15 pm
I thought thomas siad it was the start here page or something.
Well, that does not mean there can't be others...

I believe that I found and fixed the leak regarding project files. I still have to test it a couple of times before committing though.
This leak is wxWidget's fault, in my opinion. DeleteNode() suggests that this deletes the node and the data, even more so as the documentation to RemoveNode() says "you have to free the object yourself". I replaced the DeleteNode() with the code sample from the wxList documentation (basically just deletes the node and the object), and it seems that the memory leak is gone.

I get this now when opening/closing the Code::Blocks project (compiler and debugger plugin active, all others disabled, start_here page disabled)

(http://img93.imageshack.us/img93/7200/00005hy.png)

Code
Index: sdk/cbproject.cpp
===================================================================
--- sdk/cbproject.cpp (revision 1889)
+++ sdk/cbproject.cpp (working copy)
@@ -989,24 +989,18 @@
             return false;
 
  // now free the rest of the project files
-    int count = m_Files.GetCount();
     Manager::Get()->GetEditorManager()->HideNotebook();
     FilesList::Node* node = m_Files.GetFirst();
     while(node)
     {
         ProjectFile* f = node->GetData();
-        if (Manager::Get()->GetEditorManager()->Close(f->file.GetFullPath(),true))
-        {
-            FilesList::Node* oldNode = node;
-            node = node->GetNext();
-            m_Files.DeleteNode(oldNode);
-            --count;
-        }
-        else
-            node = node->GetNext();
+        Manager::Get()->GetEditorManager()->Close(f->file.GetFullPath(),true);
+        delete f;
+        delete node;
+        node = m_Files.GetFirst();
     }
     Manager::Get()->GetEditorManager()->ShowNotebook();
-    return count == 0;
+    return true;
 }
 
 bool cbProject::SaveAllFiles()
Title: Re: observation on memory consumption
Post by: thomas on January 28, 2006, 03:23:32 pm
Memory consumption now seems to be almost constant, possibly asymptotic (with code completion disabled).

With code completion enabled, the base memory still grows linearly, but the peak stays constant (that is probably an effect of the block allocator). There is also quite noticeable heap fragmentation (visible by the dull peaks and the lessened inclination of the curve after opening and closing a large project 15-20 times. I haven't had the time to test what happens when the base memory grows as high as the peak, have to go out now... :(
Title: Re: observation on memory consumption
Post by: Michael on January 28, 2006, 09:23:04 pm
Hello,

I have performed some tests with the rev1891 and I have remarked the good works of the devs&all concerning the memory leaks :D. Unfortunately, the behvior that I have observed before it is not fully gone away.

I have tried C::B without any plugins and opened/closed several times the C::B project. After closing the project, the memory is used is returned to the system, but not always and when its amount is generally less than what it was used. Sometimes the memory got accumulated and later returned to the system. This behavior was relatively chaotic and difficult to correctly describe it (sorry). Before it did not returned the memory used when opening a project (the amount was relatively constant, i.e., 1-2MB). Now, the amount of memory used to open the project is sometimes 100-200 kb, sometimes less.

IMHO, there is still something that keeps the memory and does not or not always return it to the OS.

Michael 
Title: Re: observation on memory consumption
Post by: thomas on January 28, 2006, 09:43:07 pm
Quote
IMHO, there is still something that keeps the memory and does not or not always return it to the OS.
Lol, at the risk of repeating myself... the block allocator never returns allocated memory to the OS, this is normal.

Only objects that are allocated via standard operator new are returned to the OS when they are deleted. That is for example the case for classes like wxFileName, or wxTextCtrl, but it is not true for cbEvent, Token, ProjectFile, ProjectBuildtarget, and maybe a couple of others.

Thus, it is correct that not all memory is freed, it must not be. The important point, however, is that the memory is reused, so if you open the same project 20 times (and even if you open/close different projects several times), the total amount of memory has to stay the same as long as your total memory requirement is not higher.
Title: Re: observation on memory consumption
Post by: Michael on January 28, 2006, 09:54:04 pm
Quote
IMHO, there is still something that keeps the memory and does not or not always return it to the OS.
Lol, at the risk of repeating myself... the block allocator never returns allocated memory to the OS, this is normal.

Only objects that are allocated via standard operator new are returned to the OS when they are deleted. That is for example the case for classes like wxFileName, or wxTextCtrl, but it is not true for cbEvent, Token, ProjectFile, ProjectBuildtarget, and maybe a couple of others.

Thus, it is correct that not all memory is freed, it must not be. The important point, however, is that the memory is reused, so if you open the same project 20 times (and even if you open/close different projects several times), the total amount of memory has to stay the same as long as your total memory requirement is not higher.

Thank you for your explanation :). Anyway, the total amount of memory still tends to grow (it is what I have observed). But this growth is difficult to spot, becuase it is just a matter of KB (before it was MB). 

Michael
Title: Re: observation on memory consumption
Post by: Der Meister on January 28, 2006, 10:03:20 pm
Another thing is: valgrind reported much more memory leaks than my patch solved. At least one more should be removed by thomas' patch (I did not test it yet - using valgrind needs some time; for example Code::Blocks needs about three minutes to start on my 2,4GHz Pentium 4) but others remain and I fear that we can't solve all of them. Some seem to be the fault of wxGTK or even gtk or Xlib... Anyway, as long as they are not too big and only appear once in the complete livetime of the Code::Blocks process they should not be a real problem.
Title: Re: observation on memory consumption
Post by: Michael on January 28, 2006, 10:16:22 pm
Another thing is: valgrind reported much more memory leaks than my patch solved. At least one more should be removed by thomas' patch (I did not test it yet - using valgrind needs some time; for example Code::Blocks needs about three minutes to start on my 2,4GHz Pentium 4) but others remain and I fear that we can't solve all of them. Some seem to be the fault of wxGTK or even gtk or Xlib...

Purify does the same :). The expanded report is more than 13MB (txt file). In this report are reported not only memory leaks ("real" and potential), but also null pointers, invalid pointers, exceptions, and so on related to C::B and plugins, windows libraries and wxWidgets library. With some time and patiente it would be possible to get something useful. 

Anyway, as long as they are not too big and only appear once in the complete livetime of the Code::Blocks process they should not be a real problem.

Yes, I agree. It would have no sense to try to solve all the possible memory leaks.

Michael
Title: Re: observation on memory consumption
Post by: thomas on January 28, 2006, 10:17:31 pm
The point is that the peak memory consumption does not grow any more. :)


Compare this to any of the above recordings:

(http://img453.imageshack.us/img453/1615/leak8es.png)

There are very clearly remaining leaks... there is obviously an upward tendency, but the peak allocation (which is what matters) really does not change so much.

EDIT:
It is interesting to compare the shapes of the individual spikes. You can see that the more often you open and close a project, the longer it takes to allocate memory, and there are obviously some kind of garbage collections or reorderings taking place in the middle.
I assume this is because the heap is getting so heavily fragmented with time. This was one reason why we made cbEvents and Tokens block-allocated. Those objects are constantly allocated and destroyed (many thousand times), and block-allocating them prevents heap fragmentation.
Title: Re: observation on memory consumption
Post by: Michael on January 28, 2006, 10:36:20 pm
I have done a small graphic about a fast test I have done. It looks like yours, but I think that I should have taken more sample values for a better distribution line. I will try tomorrow to do a better graph.

Michael

[EDIT] Value 1 is starting C::B, value 2 is opening C::B project, value 3 closing the workspace and so on. Vallues are expressed in KB.


[attachment deleted by admin]
Title: Re: observation on memory consumption
Post by: rickg22 on January 29, 2006, 03:59:32 am
Good news people, i got rid of that nasty fullnames tree, and the contribplugins uses max 290 MB :)
Title: Re: observation on memory consumption
Post by: Michael on January 29, 2006, 01:57:46 pm
Good news people, i got rid of that nasty fullnames tree, and the contribplugins uses max 290 MB :)

You have done a good job :D. I have tried C::B rev1893 with Code Completion (full options) and all the other plugings and it seems the C::B does not more require 100 MB for opening the C::B project. The graph attached can demonstrate it. Value 1 is starting C::B, value 2 is opening C::B project, value 3 is closing the workspace and so on. Vallues are expressed in KB.

Michael


[attachment deleted by admin]