Author Topic: Codecompletion delays fixed in rev 1826 (update: in rev 1840)  (Read 63163 times)

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« on: January 22, 2006, 02:50:07 am »
Hi all, I'm glad to announce that the long-time delays introduced in the redesign have been fixed in revision 1826 of Code::Blocks. You can use global includes again :-)

The reparsing with cache issue and the 0.5 second delay still remain, tho.
« Last Edit: January 23, 2006, 07:22:31 am by rickg22 »

sethjackson

  • Guest
Re: Codecompletion delays fixed in rev 1826 :)
« Reply #1 on: January 22, 2006, 02:51:59 am »
Bulding rev 1826 right now......  :D Now I have a long-time delay recompiling the SDK.  :P

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Codecompletion delays fixed in rev 1826 :)
« Reply #2 on: January 22, 2006, 03:20:45 pm »
Hi all, I'm glad to announce that the long-time delays introduced in the redesign have been fixed in revision 1826 of Code::Blocks. You can use global includes again :-)

Great, thanks :D.

Michael

[EDIT] After building C::B rev1828 and try opening C::B project and ContribPlugins.workspace, the speed is at least 30% higher as before. Good job :D.
« Last Edit: January 22, 2006, 04:34:39 pm by Michael »

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 :)
« Reply #3 on: January 22, 2006, 06:48:24 pm »
Startup when opening the Code::Blocks project takes 7 seconds with code completion on (about 1 second without). Total parse time is now 17 seconds instead of 40 (almost as fast as it used to be). Saving files is instantaneous.

I found two bugs regarding parsing/reparsing:
1. If you change options, you're prompted whether you want to reparse. If you change options again before that reparsing is done, and you answer "yes" another time, Code::Blocks freezes up. Mutex maybe?

2. If you change options and say "yes" to reparse, Code::Blocks will crash on exit if you close the window before the reparse is finished. Funnily, you can otherwise close the window at any time, even if the parse at startup has not finished. The crash only occurs while reparsing.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #4 on: January 23, 2006, 07:24:31 am »
I haven't fixed those bugs yet... I think.

I removed a Mutex and replaced it with a critical section, maybe that'd do the trick with the lockup.

I also removed (now confirmed :D ) the delay at startup!
Additionally, thanks to the SearchTree(TM) class (used for global filenames lookup), I could optimize the global files parsing by another 30%! :)

Next week I'll focus on the reparsing bugs.
Uh, now that I think of it... Please submit them to Sourceforge! All the newfound bugs in CC need to be organized in the database.
« Last Edit: January 23, 2006, 07:30:30 am by rickg22 »

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #5 on: January 23, 2006, 01:02:12 pm »
I removed a Mutex and replaced it with a critical section, maybe that'd do the trick with the lockup.
That's the same thing (well, for our purpose, it is), so that won't be the reason.
It rather looks as if there is no locking at all when entering the parse cycle. Apparently, two threads are concurrently modifying the same tree structure (which eventually freezes up).
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #6 on: January 24, 2006, 04:38:06 am »
Thomas: I couldn't replicate the bug. I experience a brief delay, but did as you explained, and no hang / crash was detected.

Offline tiwag

  • Developer
  • Lives here!
  • *****
  • Posts: 1196
  • sailing away ...
    • tiwag.cb
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #7 on: January 24, 2006, 10:50:47 am »
there must still be a bug in the codecompletion plugin ...

how to reproduce / what i've done:

1. create a workspace, which contains CodeBlocks and all Contrib-Plugins
2. open this workspace with CodeCompletion enabled, parse local & global includes,
    it lasts about 400sec to load and parse all projects and the memory-consumption is about 600MB !!! (->slow as molasses)
3. when i close the workspace, C::B crashes with a segmentation fault



--- for reference ---

4. open this workspace with CodeCompletion disabled, parse local & global includes,
    it lasts about 300sec to load and parse all projects and the memory-consumption is about 270MB
5. when i close the workspace, C::B doesn't crash

6. open this workspace with CodeCompletion disabled, parse local includes only,
    it lasts about 22sec to load and parse all projects and the memory-consumption is about 50MB
7. when i close the workspace, C::B doesn't crash


tested on a AMD XP 2800, 512MB RAM,
Windows XP, SP2
C::B SVN rev 1841
« Last Edit: January 24, 2006, 10:52:21 am by tiwag »

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #8 on: January 24, 2006, 11:21:02 am »
Thomas: I couldn't replicate the bug.
When closing while reparsing in 1846, I get this from Dr.MinGW:
Code
DEBUG_EVENT:
dwDebugEventCode = CREATE_PROCESS_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 714
hProcess = 718
hThread = 798
lpBaseOfImage = 400000
dwDebugInfoFileOffset = 475000
nDebugInfoSize = 426C
lpThreadLocalBase = 7FFDD000
lpStartAddress = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = CREATE_THREAD_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 220
hThread = 710
lpThreadLocalBase = 7FFDC000
lpStartAddress = 7C810856
DEBUG_EVENT:
dwDebugEventCode = CREATE_THREAD_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 214
hThread = 70C
lpThreadLocalBase = 7FFDB000
lpStartAddress = 7C810856
DEBUG_EVENT:
dwDebugEventCode = CREATE_THREAD_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 668
hThread = 708
lpThreadLocalBase = 7FFD9000
lpStartAddress = 7C810856
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 704
lpBaseOfDll = 7C910000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 700
lpBaseOfDll = 7C800000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6FC
lpBaseOfDll = 77BE0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6F8
lpBaseOfDll = 7C9D0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6F4
lpBaseOfDll = 77DA0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6F0
lpBaseOfDll = 77E50000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6EC
lpBaseOfDll = 77EF0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6E8
lpBaseOfDll = 77D10000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6E4
lpBaseOfDll = 77F40000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6E0
lpBaseOfDll = 10000000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6DC
lpBaseOfDll = 773A0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6D8
lpBaseOfDll = 76350000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6D4
lpBaseOfDll = 6FBC0000
dwDebugInfoFileOffset = 1A00
nDebugInfoSize = 19A
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6D0
lpBaseOfDll = 774B0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6CC
lpBaseOfDll = 770F0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6C8
lpBaseOfDll = 76AF0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6C4
lpBaseOfDll = 71A30000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6C0
lpBaseOfDll = 71A10000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6BC
lpBaseOfDll = 71A00000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6B8
lpBaseOfDll = 604C0000
dwDebugInfoFileOffset = BD4800
nDebugInfoSize = B0FB
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6B4
lpBaseOfDll = 8E0000
dwDebugInfoFileOffset = 3D8000
nDebugInfoSize = 2C8E
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6B0
lpBaseOfDll = 5D100000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6AC
lpBaseOfDll = 5B420000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6A8
lpBaseOfDll = 5B0F0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6A4
lpBaseOfDll = 13E0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 6A0
lpBaseOfDll = 76BB0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 69C
lpBaseOfDll = 77BD0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 698
lpBaseOfDll = 77660000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 694
lpBaseOfDll = 76F20000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 690
lpBaseOfDll = 71B70000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 68C
lpBaseOfDll = 1600000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 688
lpBaseOfDll = 76320000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 684
lpBaseOfDll = 20000000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 680
lpBaseOfDll = 4B4D0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 67C
lpBaseOfDll = 6FFC0000
dwDebugInfoFileOffset = 3C9800
nDebugInfoSize = 26C5
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 678
lpBaseOfDll = 1F90000
dwDebugInfoFileOffset = 244600
nDebugInfoSize = 2198
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 674
lpBaseOfDll = 69C00000
dwDebugInfoFileOffset = 21B000
nDebugInfoSize = 1D29
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 670
lpBaseOfDll = 64B80000
dwDebugInfoFileOffset = 61F400
nDebugInfoSize = 4E77
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 66C
lpBaseOfDll = 25E0000
dwDebugInfoFileOffset = 237400
nDebugInfoSize = 1F1E
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 668
lpBaseOfDll = 2830000
dwDebugInfoFileOffset = 3EAA00
nDebugInfoSize = 3C47
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 660
lpBaseOfDll = 2C30000
dwDebugInfoFileOffset = 18D000
nDebugInfoSize = 75B
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 65C
lpBaseOfDll = 67C40000
dwDebugInfoFileOffset = 333A00
nDebugInfoSize = 3635
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 658
lpBaseOfDll = 636C0000
dwDebugInfoFileOffset = 22BA00
nDebugInfoSize = 1E70
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 654
lpBaseOfDll = 2DD0000
dwDebugInfoFileOffset = 2B1200
nDebugInfoSize = 2622
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 650
lpBaseOfDll = 3090000
dwDebugInfoFileOffset = 3E5A00
nDebugInfoSize = 33EE
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 64C
lpBaseOfDll = 6DF40000
dwDebugInfoFileOffset = 253A00
nDebugInfoSize = 1ED7
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 648
lpBaseOfDll = 3480000
dwDebugInfoFileOffset = 27EA00
nDebugInfoSize = 2B8C
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 644
lpBaseOfDll = 6E500000
dwDebugInfoFileOffset = 236000
nDebugInfoSize = 1E06
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 640
lpBaseOfDll = 6A7C0000
dwDebugInfoFileOffset = 250A00
nDebugInfoSize = 2403
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 63C
lpBaseOfDll = 6C080000
dwDebugInfoFileOffset = D9E000
nDebugInfoSize = BE87
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 638
lpBaseOfDll = 3710000
dwDebugInfoFileOffset = 20EC00
nDebugInfoSize = 1683
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 634
lpBaseOfDll = 3AC0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 630
lpBaseOfDll = 76C50000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 62C
lpBaseOfDll = 59DD0000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = LOAD_DLL_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
hFile = 628
lpBaseOfDll = 77B10000
dwDebugInfoFileOffset = 0
nDebugInfoSize = 0
lpImageName = NULL
fUnicoded = 1
DEBUG_EVENT:
dwDebugEventCode = CREATE_THREAD_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 218
hThread = 624
lpThreadLocalBase = 7FFDA000
lpStartAddress = 7C96077B
DEBUG_EVENT:
dwDebugEventCode = EXCEPTION_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 218
ExceptionCode = 80000003
ExceptionFlags = 0
ExceptionAddress = 7C911230
dwFirstChance = 1
DEBUG_EVENT:
dwDebugEventCode = EXCEPTION_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
ExceptionCode = C0000005
ExceptionFlags = 0
ExceptionAddress = 65662D20
dwFirstChance = 0
codeblocks.exe caused an Access Violation at location 65662d20 Reading from location 65662d20.

Registers:
eax=65662d20 ebx=01295f20 ecx=025450e4 edx=00000024 esi=105f3b80 edi=0022dc24
eip=00000000 esp=0022fff8 ebp=00000000 iopl=0         nv up ei pl nz ac po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000216

Call stack:
AddrPC     AddrReturn AddrFrame  AddrStack  Params
00000000   00401240   0022FFF4   0022FFF8   00000000   78746341   00000020   00000001
00000000
DEBUG_EVENT:
dwDebugEventCode = EXIT_THREAD_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 220
dwExitCode = C0000005
DEBUG_EVENT:
dwDebugEventCode = EXIT_THREAD_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 478
dwExitCode = C0000005
DEBUG_EVENT:
dwDebugEventCode = EXIT_THREAD_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 214
dwExitCode = C0000005
DEBUG_EVENT:
dwDebugEventCode = EXIT_THREAD_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 668
dwExitCode = C0000005
DEBUG_EVENT:
dwDebugEventCode = EXIT_PROCESS_DEBUG_EVENT
dwProcessId = 70C
dwThreadId = 218
dwExitCode = C0000005

When running Code::Blocks from the debugger, it points out a segfault in GetNoteBook(). It only ever happens when closing while reparsing after changing settings. Closing the application while it is parsing otherwise works fine.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #9 on: January 24, 2006, 03:18:45 pm »
I just tried up to step 2 of what tiwag said and got Code::Blocks to die even before it finished parsing all the projects:

Quote from: Code::Blocks Debug
[08:57:18.281]: Start parsing project Code::Blocks (wx2.6)
[08:57:18.281]: Concurrent threads for pool set to 2
[09:01:20.062]: Done parsing project copystrings (314 total parsed files, 22832 tokens in 239.781 seconds).
[09:01:31.265]: Done parsing project Code Stat (322 total parsed files, 23212 tokens in 11.203 seconds).
[09:01:31.265]: Updating class browser...
[09:01:31.281]: Class browser updated.
[09:01:32.765]: Done parsing project C::B KeyBinder (328 total parsed files, 24011 tokens in 1.500 seconds).
[09:01:33.656]: Done parsing project C::B Profiler (326 total parsed files, 23375 tokens in 0.891 seconds).
[09:01:39.921]: Done parsing project Help Plugin (wx2.6) (352 total parsed files, 25091 tokens in 6.265 seconds).
[09:01:52.203]: Done parsing project Exporter (362 total parsed files, 25283 tokens in 12.282 seconds).
[09:01:54.390]: Done parsing project Dev-C++ DevPak updater/installer (wx2.6) (375 total parsed files, 26298 tokens in 2.187 seconds).
[09:02:31.203]: Done parsing project wxSmith for wx 2.6 (575 total parsed files, 29055 tokens in 36.813 seconds).

Memory usage: 263.392 KB

Something I noticed is it takes 50% CPU while parsing half of the time. It goes like 50% for a few seconds then 0% for some seconds and again.

The reason it takes 50% and not 100% is 'cause my CPU has HT, even though it's supposed to be using 2 threads...

Another funny thing is the way it crashed. The "codeblocks.exe ha detectado un problema..." window is there just reporting it crashed, but I'm still able to "use" Code::Blocks. Really, I can click on menus, the project tree, but I just tried opening the Environment settings and it is finally unresponsive :P

Since I don't compile it with debugging information it's what I get:

Quote from: Dr. MinGW
codeblocks.exe caused an Access Violation at location 7c9206c3 in module ntdll.dll Writing to location 12fffffe.

The call stack is just 00000000.

Tested on an Intel P4 3.0E, 1GiB RAM,
Windows XP, SP2
Code::Blocks SVN revision 1833

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #10 on: January 24, 2006, 03:35:23 pm »
Quote
Another funny thing is the way it crashed. The "codeblocks.exe ha detectado un problema..." window is there just reporting it crashed, but I'm still able to "use" Code::Blocks.
That must be because Code::Blocks doesn't understand Spanish, so it wouldn't know ;)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #11 on: January 24, 2006, 04:46:52 pm »
Quote from: thomas
That must be because Code::Blocks doesn't understand Spanish, so it wouldn't know :wink:

:lol:

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #12 on: January 24, 2006, 05:31:50 pm »
:shock: Agh I'm such a fool! I used static events declaration for a part of the parser, no wonder it crashes! I need to use dynamic event tables! :oops:

Now, regarding the memory consumption, each token takes around 1K of memory usage. By summing the count of all tokens, i get around 247000 tokens, including the codeblocks.cbp (i used 73000 tokens which is what I get for global includes enabled). So the codecompletion tokens alone (without including the extra memory for data structures we end up using 256Megs, and this is because many tokens are duplicated among projects.

I think the whole model needs to be redesigned, it's obviously not fit for so many large projects at once... :(

I need another favor guys. What's the token count *WITHOUT* local or global includes?
« Last Edit: January 24, 2006, 05:35:25 pm by rickg22 »

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #13 on: January 24, 2006, 05:38:14 pm »
Hello,

I have checked the memory consumption of C::B (rev1846) with code completion all enabled. When I open C::B project, the memory's requirement increases up to 92-93 MB. Some time later (5-10 secondes, but I do not know exactly. Just the time to re-open this topic and press button "reply") the memory consumption falls to 5-6 MB and the project is opened.

May be storing the search trees in the disk instead of the memory would decrease the memory's requirement...

Michael

[EDIT] And here some more info (from Code::Blocks Debug):

Quote
Done parsing project Code::Blocks (wx2.6) (980 total parsed files, 60891 tokens in 2.394 seconds).

The parsing time seems low, but another try has given me  6.229 seconds.

And without all deselected:

Quote
Done parsing project Code::Blocks (wx2.6) (542 total parsed files, 15551 tokens in 3.495 seconds).
« Last Edit: January 24, 2006, 07:19:13 pm by Michael »

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #14 on: January 24, 2006, 05:54:36 pm »
CodeBlocks.cbp for rev1833 with follow local and global includes takes 77328 KB and stays there.

Quote from: Code::Blocks Debug
Done parsing project Code::Blocks (wx2.6) (982 total parsed files, 45558 tokens in 61.109 seconds).

Now without global includes it takes 51272 KB.

Quote from: Code::Blocks Debug
Done parsing project Code::Blocks (wx2.6) (542 total parsed files, 15540 tokens in 1.766 seconds).

With both global and local disabled it takes 51332 KB and has the same results about parsed files and tokens found.

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #15 on: January 24, 2006, 06:09:19 pm »
Hmmm... I have slightly different figures.

Using this workspace:
Code
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_workspace_file>
<Workspace title="All contrib plugins">
<Project filename="cb\plugins\contrib\codestat\codestat.cbp" active="1" />
<Project filename="cb\plugins\contrib\copystrings\copystrings.cbp" />
<Project filename="cb\plugins\contrib\devpak_plugin\DevPakPlugin.cbp" />
<Project filename="cb\plugins\contrib\help_plugin\help-plugin.cbp" />
<Project filename="cb\plugins\contrib\keybinder\keybinder.cbp" />
<Project filename="cb\plugins\contrib\profiler\cbprofiler.cbp" />
<Project filename="cb\plugins\contrib\source_exporter\Exporter.cbp" />
<Project filename="cb\plugins\contrib\wxSmith\wxSmith.cbp" />
<Project filename="cb\CodeBlocks.cbp" />
</Workspace>
</CodeBlocks_workspace_file>

I get 494.2 MB (with code completion) versus 12.6 MB (no code completion).

[attachment deleted by admin]
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #16 on: January 24, 2006, 06:16:30 pm »
And here are the token counts.

Everything turned on:
Code
[18:12:07.281]: Start parsing project Code Stat
[18:12:07.281]: Concurrent threads for pool set to 1
[18:12:08.125]: Start parsing project copystrings
[18:12:08.125]: Concurrent threads for pool set to 1
[18:12:08.609]: Start parsing project Dev-C++ DevPak updater/installer (wx2.6)
[18:12:08.609]: Concurrent threads for pool set to 1
[18:12:09.765]: Start parsing project Help Plugin (wx2.6)
[18:12:09.765]: Concurrent threads for pool set to 1
[18:12:10.390]: Start parsing project C::B KeyBinder
[18:12:10.390]: Concurrent threads for pool set to 1
[18:12:11.109]: Start parsing project C::B Profiler
[18:12:11.109]: Concurrent threads for pool set to 1
[18:12:11.656]: Start parsing project Exporter
[18:12:11.656]: Concurrent threads for pool set to 1
[18:12:12.437]: Start parsing project wxSmith for wx 2.6
[18:12:12.437]: Concurrent threads for pool set to 1
[18:12:13.421]: Start parsing project Code::Blocks (wx2.6)
[18:12:13.421]: Concurrent threads for pool set to 1
[18:12:46.687]: Done parsing project Help Plugin (wx2.6) (550 total parsed files, 57254 tokens in 27.953 seconds).
[18:12:52.953]: Done parsing project C::B Profiler (514 total parsed files, 53061 tokens in 6.266 seconds).
[18:12:54.843]: Done parsing project Code Stat (506 total parsed files, 52749 tokens in 1.890 seconds).
[18:12:54.843]: Updating class browser...
[18:12:54.859]: Class browser updated.
[18:12:57.140]: Done parsing project Dev-C++ DevPak updater/installer (wx2.6) (536 total parsed files, 53977 tokens in 2.297 seconds).
[18:12:57.218]: Done parsing project Exporter (566 total parsed files, 57653 tokens in 0.78 seconds).
[18:12:58.531]: Done parsing project C::B KeyBinder (524 total parsed files, 55687 tokens in 1.313 seconds).
[18:13:00.031]: Done parsing project copystrings (573 total parsed files, 57413 tokens in 1.500 seconds).
[18:13:10.171]: Done parsing project wxSmith for wx 2.6 (841 total parsed files, 65030 tokens in 10.140 seconds).
[18:13:16.718]: Done parsing project Code::Blocks (wx2.6) (1210 total parsed files, 79395 tokens in 6.547 seconds).

Everything turned off:
Code
[18:15:16.468]: Start parsing project Code Stat
[18:15:16.468]: Concurrent threads for pool set to 1
[18:15:16.781]: Start parsing project copystrings
[18:15:16.781]: Concurrent threads for pool set to 1
[18:15:16.937]: Done parsing project Code Stat (7 total parsed files, 35 tokens in 0.0 seconds).
[18:15:16.937]: Updating class browser...
[18:15:16.937]: Class browser updated.
[18:15:17.078]: Start parsing project Dev-C++ DevPak updater/installer (wx2.6)
[18:15:17.078]: Concurrent threads for pool set to 1
[18:15:17.234]: Done parsing project copystrings (2 total parsed files, 9 tokens in 0.0 seconds).
[18:15:17.515]: Start parsing project Help Plugin (wx2.6)
[18:15:17.515]: Concurrent threads for pool set to 1
[18:15:17.687]: Done parsing project Dev-C++ DevPak updater/installer (wx2.6) (24 total parsed files, 348 tokens in 0.16 seconds).
[18:15:17.828]: Start parsing project C::B KeyBinder
[18:15:17.828]: Concurrent threads for pool set to 1
[18:15:18.062]: Done parsing project Help Plugin (wx2.6) (6 total parsed files, 59 tokens in 0.78 seconds).
[18:15:18.125]: Start parsing project C::B Profiler
[18:15:18.125]: Concurrent threads for pool set to 1
[18:15:18.296]: Done parsing project C::B KeyBinder (6 total parsed files, 396 tokens in 0.15 seconds).
[18:15:18.437]: Start parsing project Exporter
[18:15:18.437]: Concurrent threads for pool set to 1
[18:15:18.671]: Done parsing project C::B Profiler (6 total parsed files, 47 tokens in 0.78 seconds).
[18:15:18.875]: Start parsing project wxSmith for wx 2.6
[18:15:18.875]: Concurrent threads for pool set to 1
[18:15:19.062]: Done parsing project Exporter (21 total parsed files, 713 tokens in 0.78 seconds).
[18:15:19.234]: Start parsing project Code::Blocks (wx2.6)
[18:15:19.234]: Concurrent threads for pool set to 1
[18:15:19.656]: Done parsing project wxSmith for wx 2.6 (196 total parsed files, 3033 tokens in 0.266 seconds).
[18:15:24.312]: Done parsing project Code::Blocks (wx2.6) (547 total parsed files, 15636 tokens in 1.641 seconds).
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #17 on: January 24, 2006, 06:23:11 pm »
When you say "Everything turned on" you mean "Parse preprocessor directives too?".

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #18 on: January 24, 2006, 06:24:24 pm »
Yep. Everything. :)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #19 on: January 24, 2006, 06:35:07 pm »
Quote from: Rick
I think the whole model needs to be redesigned, it's obviously not fit for so many large projects at once... :(

OK assumming preprocessor directives take 5% of the total token count, and that the wxWidgets project gives us around 55,000 tokens... the sum of distinct tokens for all the workspace gives us a total of less than 90,000 tokens. And the parsing time would be less than 20 seconds.

Argh... I need to consult this with Yiannis.
« Last Edit: January 24, 2006, 07:08:14 pm by rickg22 »

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #20 on: January 24, 2006, 06:58:35 pm »
Trying again with revision 1833 and everything on using Thomas' workspace.

Quote from: Code::Blocks Debug
[12:46:51.390]: Start parsing project Code::Blocks (wx2.6)
[12:46:51.390]: Concurrent threads for pool set to 2
[12:46:53.656]: Running startup script
[12:50:57.687]: Done parsing project copystrings (372 total parsed files, 45288 tokens in 244.0 seconds).
[12:51:00.234]: Done parsing project Code Stat (381 total parsed files, 45760 tokens in 2.547 seconds).
[12:51:00.234]: Updating class browser...
[12:51:00.265]: Class browser updated.
[12:51:11.421]: Done parsing project C::B KeyBinder (390 total parsed files, 46579 tokens in 11.187 seconds).
[12:51:12.484]: Done parsing project C::B Profiler (386 total parsed files, 45927 tokens in 1.63 seconds).
[12:51:30.593]: Done parsing project Help Plugin (wx2.6) (414 total parsed files, 47788 tokens in 18.109 seconds).
[12:51:31.578]: Done parsing project Exporter (423 total parsed files, 47872 tokens in 0.985 seconds).
[12:51:33.703]: Done parsing project Dev-C++ DevPak updater/installer (wx2.6) (441 total parsed files, 49220 tokens in 2.125 seconds).
[12:52:10.656]: Done parsing project wxSmith for wx 2.6 (644 total parsed files, 52045 tokens in 36.953 seconds).
[12:52:50.703]: Done parsing project Code::Blocks (wx2.6) (1083 total parsed files, 73764 tokens in 40.47 seconds).

It didn't crash this time!



Take a look at the CPU usage. Can you see what I meant?

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #21 on: January 24, 2006, 06:59:12 pm »
Quote
And the parsing time would be less than 20 seconds.

Below 20 seconds?  :?
18:12:07 -- 18:13:16 = 69 seconds.
69 seconds * 0,95 = 65 seconds (more or less).

This is a fully optimized build, by the way (-DNDEBUG -O3 -fexpensive-optimizations -march=k8 -ffast-math). Debug build takes about twice as long.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #22 on: January 24, 2006, 07:06:45 pm »
What I mean is that it would take 20 seconds if we decouple the tokens / files, so no file would be parsed more than once. This would imply that there would be one global parser, instead of one per project. It would also allow us to make a transition to a database if it was necessary.

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #23 on: January 24, 2006, 07:08:32 pm »
Quote
Take a look at the CPU usage. Can you see what I meant?
Yes, and there is something else (besides the hyperthreading thing) to note. You can very clearly see that max_threads = NUM_CPUS is not a good choice.
Look at the regular "zero CPU" intervals, the CPU is stalled 50% of the time (very likely waiting for an I/O operation to finish). So, overall, you use only 25% of your CPU's capacity. It could very well do using two more threads, so one thread runs while another waits for I/O.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #24 on: January 24, 2006, 09:36:05 pm »
So you recommend adding 1 or 2 extra threads?

Hmm interesting :)

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #25 on: January 24, 2006, 09:47:33 pm »
Well, look at the screenshot. The CPU is completely idle 50% of the time (and runs one thread during the other half, when it could run two).

An even better solution might be to have exactly one dedicated I/O thread which does all reads asynchronously, but sequentially and feeds the results to the blocking NUM_CPU+1 worker threads.
Under Linux, this probably won't bring an awful lot, since the elevator mechanism should already make sure that the hard disk is not trashing when two threads try to read concurrently, but I wouldn't be so sure about that under Windows...

If someone can produce a cross-platform (posix + Windows) memory mapping class, then we might leave that work to the OS, too. It would probably be the most efficient way, anyway.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #26 on: January 24, 2006, 10:26:35 pm »
Just a suggestion. Would be of some interest the use of Win32 pthreads library in Windows? It was used to develop e.g., ViTooKi (a cross-platform C++ multimedia library).

Michael
 

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #27 on: January 24, 2006, 10:45:08 pm »
Just a suggestion. Would be of some interest the use of Win32 pthreads library in Windows?
Win32 pthreads does not offer memory mapping, unluckily. The functionality which it offers is in wxWidgets too, so I don't think we can gain a lot from using it.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Urxae

  • Regular
  • ***
  • Posts: 376
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #28 on: January 24, 2006, 11:56:10 pm »
If someone can produce a cross-platform (posix + Windows) memory mapping class, then we might leave that work to the OS, too. It would probably be the most efficient way, anyway.

Dare I suggest... Boost has such a class, in <boost/spirit/iterator/file_iterator.hpp>. It provides an iterator interface to files on disk.
There are basically 3 implementations (the appropriate one is selected using the preprocessor): One for Win32, one for Posix, and one using only the C++ standard library. The first two implement memory mapping, the other one is provided as a fallback.
If you don't want to include all of Boost though (which is understandable) you'll either have to check exactly which files are needed (and/or edit it to reduce dependencies manually) or perhaps just use it as an example for your own implementation.

The Boost license is GPL compatible, IIRC.

Offline Game_Ender

  • Lives here!
  • ****
  • Posts: 551
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #29 on: January 25, 2006, 01:21:07 am »
The boost license is about a loose as it gets you just need to basically include the copyright from the author and that's about it.  Boost has a good thread library too.  If you are going to use an external thread library I would use boost, because then you can use lots of the other wonderful stuff boost has.

As for this I have one question, why are we using threads at all if doesn't run more than one at once?  I think you mean it only runs one <i>extra</i> thread, counting the main thread if you have one CPU?, or it only one thread total include the main one? The whole point of threads is to run more than the number of processors you have because the CPU is rarely full on, threads of execution spend quite a bit of time waiting.  Also I think the threading model needs a thorough rewrite or going over because it dead locks all the time on linux.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #30 on: January 25, 2006, 01:52:16 am »
"One extra thread" the answer is. More refinement Code completion needs, indeed. Hmm.

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #31 on: January 25, 2006, 03:05:13 am »
I wonder if setting CodeCompletion to use only 1 thread would "help". It's like if instead of using 2 threads it was using 1/2 thread.

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #32 on: January 25, 2006, 09:01:43 am »
As for this I have one question, why are we using threads at all if doesn't run more than one at once?
It does not freeze the GUI for 20 seconds.

Quote
I think you mean it only runs one <i>extra</i> thread, counting the main thread if you have one CPU?, or it only one thread total include the main one?
The former is the case (obviously, nothing else makes sense).
You could indeed argue why the "thread pool" is rather a "thread blocker" than a real thread pool, but that is not the cause of the bottlenecks in parsing. Rick initially wrote that code to be simple, and he is certainly aware of the unnecessary thread creation overhead. That, however, is not the thing which causes all the problems which we are facing in code completion. Since we are only creating a dozen or two unnecessary threads, this is manageable, even on such a bad performer like Windows. He might change that one day, but only after the real issues are solved.

Quote
The whole point of threads is to run more than the number of processors you have because the CPU is rarely full on, threads of execution spend quite a bit of time waiting.
Look at Ceniza's screenshot. It has a higher resolution than mine, so the issue is better visible there.
His CPU is 75% idle on the average (if you count in hyperthreading). 50% of the time, it uses one CPU, and 50% of the time, it is completely stalled. Thus, he could as well run two more concurrent threads without any penalites.

Quote
Also I think the threading model needs a thorough rewrite or going over because it dead locks all the time on linux.
Yes, although the deadlocks do not necessarily come from that, the parser does some heavy locking here and there too, so it might as well be that.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #33 on: January 25, 2006, 11:30:42 am »
I tried revision 1854 last night which creates 4 threads and I have the same results :(

BTW, those times (working then stalled) are about 3-5 seconds each (haven't timed them though, but they really count in seconds). Doesn't look like waiting for I/O to me.

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #34 on: January 25, 2006, 11:36:21 am »
Uh... seconds? What on earth is taking seconds? (I was thinking you had the task monitor running at higher speed... :lol:)

Rick, you don't happen to trigger the next parser with a wxTimer event, do you?
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #35 on: January 25, 2006, 11:42:11 am »
By checking the process' threads you can see all of them are executed in the working time (I got 36 threads with Thomas' workspace: 9 projects, 4 threads per project). When the CPU is stalled... well... you know... they do nothing.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #36 on: January 25, 2006, 05:50:26 pm »
Rick, you don't happen to trigger the next parser with a wxTimer event, do you?

Nope... this is how it works: The wxTimer works as a "countdown for start parsing." When a new file is requested to be parsed (from the main thread, that is), it resets the countdown. When all files are requested, the countdown begins and finally the parsing is triggered. This is to avoid problems with adding files recursively taking forever.

But I think the part that measures the time is broken :P I was going to change it, but i was too tired last night.

But don't worry! Everytime we're getting closer to a bug free plugin :)

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #37 on: January 25, 2006, 06:29:23 pm »
Rick: could you add the textbox for the concurrent threads again? I'd like to try it with only 1 thread.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #38 on: January 25, 2006, 07:59:07 pm »
Um... :oops: I don't know how... I didn't touch that code :P

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #39 on: January 28, 2006, 12:18:13 am »
I'm trying the latest changes in CodeCompletion and the results are very similar: 50% working time and about 60-75% CPU usage (when working).

Even better, Code::Blocks just died with a "Reading from 00000000" error.

Revision 1889.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #40 on: January 28, 2006, 12:23:52 am »
<i>Even better, Code::Blocks just died with a "Reading from 00000000" error.</i>

If you told me the exact address of the fault (a line # would be even better), i'd gladly appreciiate it.

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #41 on: January 28, 2006, 01:28:49 am »
Sorry Rick but the report isn't good enough to point you somewhere since I always compile with -s -O3 instead of -g.

I'll rebuild it again with -g and try the same test. If I get something I'll tell you.

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Codecompletion delays fixed in rev 1826 (update: in rev 1840)
« Reply #42 on: January 28, 2006, 01:51:30 am »
Quote from: gdb
Program received signal SIGSEGV, Segmentation fault.
[Switching to thread 2772.0x4ac]
0x77c17026 in msvcrt!memcpy () from C:\WINDOWS\system32\msvcrt.dll
(gdb) bt
#0  0x77c17026 in msvcrt!memcpy () from C:\WINDOWS\system32\msvcrt.dll
#1  0x10063346 in wxStringBase::ConcatSelf ()
   from C:\WINDOWS\system32\wxmsw26u_gcc_custom.dll
#2  0x65ed1cf8 in wxStringBase::ConcatSelf (this=0x14e0ed8c, nLen=239120052,
    src=0x1bd2dac) at C:/MinGW/include/wx/string.h:281
#3  0x65ed1d96 in wxStringBase::append (this=0x14e0ed8c, str=@0x651fc78)
    at C:/MinGW/include/wx/string.h:400
#4  0x65ed6438 in wxString::append (this=0x14e0ed8c, str=@0x651fc78)
    at C:/MinGW/include/wx/string.h:1157
#5  0x65ed66fc in wxString::operator<< (this=0x14e0ed8c, s=@0x651fc78)
    at C:/MinGW/include/wx/string.h:889
#6  0x65eab056 in ParserThread::DoParse (this=0x14e0ed20)
    at plugins/codecompletion/parser/parserthread.cpp:654
#7  0x65eacc4f in ParserThread::HandleClass (this=0x14e0ed20, isClass=true)
    at plugins/codecompletion/parser/parserthread.cpp:1017
#8  0x65eaa51b in ParserThread::DoParse (this=0x14e0ed20)
    at plugins/codecompletion/parser/parserthread.cpp:518
#9  0x65ea9c00 in ParserThread::Parse (this=0x14e0ed20)
    at plugins/codecompletion/parser/parserthread.cpp:372
#10 0x65ed1695 in ParserThread::Execute (this=0x14e0ed20)
    at plugins/codecompletion/parser/parserthread.h:43
#11 0x61965fa9 in PrivateThread::Entry (this=0x2c051c8)
    at sdk/cbthreadpool.cpp:86
...