Author Topic: observation on memory consumption  (Read 65538 times)

Offline tiwag

  • Developer
  • Lives here!
  • *****
  • Posts: 1196
  • sailing away ...
    • tiwag.cb
observation on memory consumption
« on: January 26, 2006, 08:52:48 pm »
what i've done:

codecompletion plugin settings:
codecompletion: enabled,
parser: local includes and preprocessor definitions

i started C::B, memory settled at 13MB
loaded CodeBlocks.cbp project, memory settled at 36MB (peak is somewhat higher)
closed project, memory settled at 23MB, #note: + 10MB
loaded CodeBlocks.cbp project, memory settled at 37MB,
....

[edit] typos "memeory"

so we get a series of

13-36-23-37-24-38-26-40-27-42-28-43-44-32-47-33  [MB] memory consumption only by opening/closing the project

mean value (without first +10MB rise) :  delta-memory/project-load = (33 -23)MB / 7 = ~1.43MB leaked memory
and this is proportional to the amount of memory which is needed for all of the parsed symbols.

( hint: those with 2GB of ram, try it with local & global includes switched on ;-)  )

[attachment deleted by admin]
« Last Edit: January 26, 2006, 08:57:48 pm by tiwag »

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #1 on: January 26, 2006, 09:10:28 pm »
Hmm... this measurement is inaccurate in one point.

Parser tokens are blockallocated objects, so that memory is never really freed. The application retains the memory block even if the objects are freed and reuses it when the next project is opened. So it appears to be leaking even if everything works correctly.

The same thing is true for ProjectFile objects, and ProjectBuildTargets, which are both block allocated, too.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: observation on memory consumption
« Reply #2 on: January 26, 2006, 09:12:21 pm »
OK Tiwag, try removing the blockallocated stuff and see if there's a leak. Is the leak still there? What happens if we also remove the codecompletion plugin?

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #3 on: January 26, 2006, 09:17:22 pm »
No need to remove the block allocator. There *is* a leak for sure.

I only said that the measurement is not accurate because of the block allocator, as it will always retain some memory. However, that memory does not grow and grow and grow indefinitely if you keep closing and re-opening a project. It always stays the same.

You may want to enable debug mode in the block allocator if you are interested whether it's Tokens what is leaked...
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #4 on: January 26, 2006, 09:58:43 pm »
Here is a more detailled explanation of why the block allocator can confuse you.

These are two recordings of the very same actions using the very same program and the same projects, but in different order:



1. Code::Blocks starts
2. Open wxSmith.cbp
3. Close wxSmith.cbp (here you get the idea "hey, what's happening, memory is leaked?")
4. Open CodeBlocks.cbp
5. Close CodeBlocks.cbp (again memory seems to be leaked)
6. Open wxSmith.cbp again (now it seems to consume no memory at all!)

Let's try the reverse order:

A. Code::Blocks starts
B. Open CodeBlocks.cbp
C. Close CodeBlocks.cbp
D. Open wxSmith.cbp

Again, opening and closing CodeBlocks.cbp seems to massively leak memory, but wxSmith.cbp does not seem to consume any memory at all.

This phenomenon is caused by the fact that the block allocator retains the memory and reuses it for the newly allocated objects. :)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: observation on memory consumption
« Reply #5 on: January 26, 2006, 11:03:34 pm »
280: Where are your patches?

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: observation on memory consumption
« Reply #6 on: January 27, 2006, 12:15:53 am »
I don't care, just show us which lines are at fault...

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #7 on: January 27, 2006, 01:20:19 am »
Rick, you will be delighted to hear that the memory leak has nothing to do with code completion :)
Code completion takes up so much RAM that the actual leaks (which are a lot smaller) are not immediately visible at all.

The most surprising observation when sampling the application's memory without code completion is that the amount of allocated memory actually rises by a good amount when you close a project!

I have been quite puzzled by that observation for a while, but I found out why: It is the start here page.
Apparently, a new page is allocated every time it is shown (and it is never freed). I'll look into the code tomorrow. Too tired to do that now, wouldn't want to break something... :)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: observation on memory consumption
« Reply #8 on: January 27, 2006, 01:32:04 am »
Yay! Thanks! :)

Offline Game_Ender

  • Lives here!
  • ****
  • Posts: 551
Re: observation on memory consumption
« Reply #9 on: January 27, 2006, 03:03:40 am »
Do any of the three core developers run Codeblocks on linux routinely?  If they do I would suggested putting Codeblocks through valgrind, I will try to do this tonight. My guess is it won't be so pretty.  One problem though is that wxWidgets itself has some small memory allocation errors, which happers the error finding processs. 

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: observation on memory consumption
« Reply #10 on: January 27, 2006, 06:04:31 am »
sorry, i don't... :(

by the way, i updated Codecompletion again today.

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: observation on memory consumption
« Reply #11 on: January 27, 2006, 12:48:42 pm »
Hello,

I have done some test with C::B (rev1884) about memory and remarked the following. At the start up, C::B memory requirement goes up to 27-28 MB. When I open the C::B project, C::B memory requirement goes up to 121-122 MB (with code completion). When closing the project (by using "close workspace"), the 50% of the memory used is returned. The same behavior happen without code completion, only that the used memory is much less.

IMHO there are some memory problems with both C::B itself and with code completion plugin.

Michael

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: observation on memory consumption
« Reply #12 on: January 27, 2006, 01:16:01 pm »
When closing the project (by using "close workspace"), the 50% of the memory used is returned. The same behavior happen without code completion, only that the used memory is much less.
This is normal behaviour. About 50% of the memory belongs to block allocated objects, these are never freed. Thus it always looks like there is a massive leak at first glance.

You cannot find out about memory leaks as simple as this (see my above post with the memory diagram). If you want to detect a memory leak by observing the total memory consumption, then you have to open/close the same project many times (10-20 times) as tiwag did.
If there are no memory leaks, you will still see that only 50% of the memory is freed, but if there is a leak (as it is the case), 50% of the peak will be freed every time, but the base will grow linearly.

Other options would be to
a) Compile wxWidgets in debug mode and run Code::Blocks from console. This will detect leaks of non-blockallocated objects.

b) Recompile Code::Blocks using the block allocator in debug mode. This will detect leaks of blockallocated objects. To do this, pass a positive integer as an additional template parameter to the classes which you want to observe:
  class Token  : public BlockAllocated<Token, 10000>
becomes
  class Token  : public BlockAllocated<Token, 10000, 1>

c) Use valgrind, but it is a lot easier to recompile wxWidgets in debug mode
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: observation on memory consumption
« Reply #13 on: January 27, 2006, 02:07:03 pm »
When closing the project (by using "close workspace"), the 50% of the memory used is returned. The same behavior happen without code completion, only that the used memory is much less.
This is normal behaviour. About 50% of the memory belongs to block allocated objects, these are never freed. Thus it always looks like there is a massive leak at first glance.

You cannot find out about memory leaks as simple as this (see my above post with the memory diagram). If you want to detect a memory leak by observing the total memory consumption, then you have to open/close the same project many times (10-20 times) as tiwag did.
If there are no memory leaks, you will still see that only 50% of the memory is freed, but if there is a leak (as it is the case), 50% of the peak will be freed every time, but the base will grow linearly.

Thank you for your answer :). I have not open/close the same project 10-20 times, but it is sometimes already that I check C::B memory consumption when opening/closing the C::B project. Interesting is that after closing it and do some work with another application (e.g., IE, Word) and later checking the memory consumption of C::B, the value is of only 2-4 MB.

Anyway, I have tried C::B with Purify. You can find the report(s) in this topic.

Michael


Offline tiwag

  • Developer
  • Lives here!
  • *****
  • Posts: 1196
  • sailing away ...
    • tiwag.cb
Re: observation on memory consumption
« Reply #14 on: January 27, 2006, 02:08:16 pm »
Quote
... but it is a lot easier to recompile wxWidgets in debug mode

to build CB with wx-debug libs is easy, but it doesn't run,

i've done this already but couldn't solve the never ending assertions ...

it seems, that any eventhandler inside CB throws assertions all the time - if this could be solved, we can debug the memory leaks