Code::Blocks Forums

User forums => Nightly builds => Topic started by: killerbot on September 27, 2010, 06:08:14 pm

Title: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on September 27, 2010, 06:08:14 pm
Get quick announcements through the RSS feed http://www.codeblocks.org/nightly/CodeBlock_RSS.xml

Before you use a nightly make sure you understand how it works (http://forums.codeblocks.org/index.php/topic,3232.0.html).

A link to the unicode windows wxWidget dll for Code::Blocks : http://prdownload.berlios.de/codeblocks/wxmsw28u_gcc_cb_wx2810_gcc441.7z

For those who might need this one (when no MingW installed on your system) : the mingw10m.dll : http://prdownload.berlios.de/codeblocks/mingwm10_gcc441.7z

The 25 September 2010 build is out.
  - Windows :
   http://prdownload.berlios.de/codeblocks/CB_20100925_rev6634_CC_BRANCH_win32.7z
  - Linux :
   none

Important changes compared to previous CODECOMPLETION BRANCH nightly:

* cc_branch: applied patch to fix a crash under Windows and do some cleanup, namely:
- fix a crash under windows
- use member variables with the unified naming style
- now NativeParser::GetParser returns always a reference
- removing parser instance unnecessary
- parser for the active editor is never removed
- refactoring some variable names, function names
* cc_branch: applied patch to fix *NONE* project parser error AND add parsing any files opened through DDE or the command-line
* cc_branch: applied patch to fix a hang while reparsing
* cc_branch: applied patch to fix an error when opening a header file without a project
* cc_branch: applied patch to improve locker for system headers
* cc_branch: applied patch simplify interface to batch parser AND improve performance
* cc_branch: applied patch to support CC for member variable initialisation
* cc_branch: build fix for parser test project
* cc_branch: reverted patch of 6238r to fix the issue reported here: http://forums.codeblocks.org/index.php/topic,13338.msg89975.html#msg89975
* cc_branch: applied patch to fix system headers parsing completely hangs
* cc-branch: reverted commit 6629 (re-revert commit 6238), it was not the cause for the issuedescribed in http://forums.codeblocks.org/index.php/topic,13338.msg89975.html#msg89975;
partly reverted commit 6594, that causes the issue
* cc_branch: applied patch to add some more comments AND improve consistency of style for header files
* cc_branch: little cosmetic fix (comparison with pointer)
* cc_branch: applied patch to add possibility to remove all bookmarks
* all updates that occurred on trunk

THIS IS A SPECIAL TEST BUILD OF REFACTORINGS CARRIED OUT ON THE CODE COMPLETION BRANCH IN OUR SVN.
FOCUS IS ON ENHANCED CODE COMPLETION USABILITY.

Give your feedback on this version only in this thread, don't mix it with the regular nightly please.

Once we don't have any blockers on this version,we will merge the changes into trunk and it will be part of the regular nightlies.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on September 27, 2010, 08:49:41 pm
Debian packages (binaries and sources) for 32-bit and 64-bit systems can be found in my repo.

If you want to use apt (or dselect, synaptic or whatever) you need to add the following entries to /etc/apt/sources.list :
Code
deb http://apt.jenslody.de/ any cc
deb-src http://apt.jenslody.de/ any cc
and remove entries for the normal nightlies.

Alternatively you can download the deb's directly from http://apt.jenslody.de/pool/cc/c/codeblocks/ (http://apt.jenslody.de/pool/cc/c/codeblocks/) .
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: superbaby on September 28, 2010, 02:15:12 am
Where is the download link for Windows?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: MortenMacFly on September 28, 2010, 07:07:23 am
Where is the download link for Windows?
In the first post by killerbot.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Borr on September 28, 2010, 07:57:43 am
Where is the download link for Windows?
In the first post by killerbot.

Quote
The 19 September 2010 build is out.
  - Windows :
   http://prdownload.berlios.de/codeblocks/CB_20100919_rev6608_CC_BRANCH_win32.7z
  - Linux :
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on September 28, 2010, 08:13:52 am
Please replace from 6608 to 6634, the fixed download link is:
http://prdownload.berlios.de/codeblocks/CB_20100919_rev6634_CC_BRANCH_win32.7z (http://prdownload.berlios.de/codeblocks/CB_20100919_rev6634_CC_BRANCH_win32.7z)

@killerbot could you fix the download url error?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: danteri on September 28, 2010, 08:14:18 am
it should be

http://prdownload.berlios.de/codeblocks/CB_20100925_rev6634_CC_BRANCH_win32.7z

Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: superbaby on September 28, 2010, 10:31:27 am
it should be

http://prdownload.berlios.de/codeblocks/CB_20100925_rev6634_CC_BRANCH_win32.7z


thanks.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on September 28, 2010, 10:56:34 am
fixed in the original post. Thanks guys.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on September 30, 2010, 05:05:35 am
1. Fix batch parse failed when create a empty project
2. Improve AddParseThread
3. Make code clean of "token.h"
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: polygon7 on September 30, 2010, 12:24:30 pm
Hi,
I have following segfaults when loading some big project ~800k LOC /
~3000-4000 files (but about 10 files are opened) with C::B version 10.05cc6634-1
from Jens repository (CC branch). I have it usually three times on five C::B starts.

Quote
Editor Open
project data set for /home/xxxxx.h
Top Editor: /home/xxxxx.cpp

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
[New Thread 0xa9bb1b70 (LWP 30027)]

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
Caching GCC dir: /usr/include/c++/4.4
Caching GCC dir: /usr/include/c++/4.4/i486-linux-gnu
Caching GCC dir: /usr/include/c++/4.4/backward
Caching GCC dir: /usr/local/include
Caching GCC dir: /usr/lib/gcc/i486-linux-gnu/4.4.3/include
Caching GCC dir: /usr/lib/gcc/i486-linux-gnu/4.4.3/include-fixed
Caching GCC dir: /usr/include
Passing list of files to batch-parser.
Header to parse up-front: '/usr/include/c++/4.4/cstddef'
Add up-front parsing 1 file(s) for project 'xxxxx'...
Add batch-parsing 4979 file(s) for project 'xxxxx'...
Create new parser for project 'xxxxx'
Starting batch parsing for project 'xxxxx'...
Start switch from OnParserStart::ptCreateParser
Switch parser to project 'xxxxx'
[New Thread 0xa93b0b70 (LWP 30032)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xa9bb1b70 (LWP 30027)]
0x00000adf in ?? ()
(gdb) bt
#0  0x00000adf in ?? ()
#1  0x00f06698 in wxEvtHandler::AddPendingEvent(wxEvent&) () from /usr/lib/libwx_baseu-2.8.so.0
#2  0x00263e91 in wxPostEvent (this=0xb1c148c) at /usr/include/wx-2.8/wx/event.h:2570
#3  cbThreadPool::TaskDone (this=0xb1c148c) at cbthreadpool.cpp:167
#4  0x002642e4 in cbThreadPool::cbWorkerThread::Entry (this=0xa936000) at cbthreadpool.cpp:236
#5  0x00f058b8 in wxThreadInternal::PthreadStart(wxThread*) () from /usr/lib/libwx_baseu-2.8.so.0
#6  0x00f0591d in wxPthreadStart () from /usr/lib/libwx_baseu-2.8.so.0
#7  0x00f7396e in start_thread (arg=0xa9bb1b70) at pthread_create.c:300
#8  0x01193a4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
(gdb)

Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on September 30, 2010, 01:02:49 pm
What is the meaning about this log?
Quote
** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
[New Thread 0xa9bb1b70 (LWP 30027)]

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
:(
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: polygon7 on September 30, 2010, 01:53:32 pm
What is the meaning about this log?
Quote
** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
[New Thread 0xa9bb1b70 (LWP 30027)]

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
:(

I think it is related to gtk/wxgtk, but unfortunately google says nothing special about that message.
I don't remember if I see it earlier but I don't run C::B in console or in gdb very often.


// EDIT: Yet another thing - I have lot of messages:
Quote
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...

and Ctrl+Space (CC) is not working.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: childinsilence on September 30, 2010, 02:03:54 pm
I think it is related to gtk/wxgtk, but unfortunately google says nothing special about that message.
I don't remember if I see it earlier but I don't run C::B in console or in gdb very often.

The function can be found here (for example, don't know which version you use): Google Codesearch (http://www.google.com/codesearch/p?hl=en#enJBsFkqeyI/sources/gtk-engines/2.9/gtk-engines-2.9.0.tar.gz|imiUVQuppLo/gtk-engines-2.9.0/engines/smooth/src/engine/smooth_gtk2_drawing.c&q=smooth_draw_flat_box)

The message you get is related to GTK, because I get it in one of my wxWidget applications too.

EDIT: Forgot to say, that I use wxGTK on the one and wxMSW on another system. I get that message only on wxGTK.  :lol:
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on September 30, 2010, 04:51:57 pm
I found some issues with code completion. They are described here : http://forums.codeblocks.org/index.php/topic,13413.0.html
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 01, 2010, 07:37:43 am
What is the meaning about this log?
Quote
** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
[New Thread 0xa9bb1b70 (LWP 30027)]

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
:(

I think it is related to gtk/wxgtk, but unfortunately google says nothing special about that message.
I don't remember if I see it earlier but I don't run C::B in console or in gdb very often.

I don't get it here (debian 64-bit unstable/experimental mix).
Errors like this are often caused by gtk2-changes.
Which OS are you on and what is the version of libgtk2 ?
 
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 01, 2010, 08:01:43 am
Hi,
I have following segfaults when loading some big project ~800k LOC /
~3000-4000 files (but about 10 files are opened) with C::B version 10.05cc6634-1
from Jens repository (CC branch). I have it usually three times on five C::B starts.

I don't get a segfault, but cc seems to either hang in a loop or do nothing (C::B is responsible), but it starts parsing and I never get a message that parsing ends.
Do nothing is not correct, because it uses cpu-power and memory (see below).

I also do not get any entries in symbols-browser.
These are the last lines of the debug-output:
Passing list of files to batch-parser.
Code
Header to parse up-front: '/usr/include/c++/4.4/cstddef'
Header to parse up-front: '/usr/include/boost/config.hpp'
Header to parse up-front: '/usr/include/boost/filesystem/config.hpp'
Add up-front parsing 3 file(s) for project 'test'...
Add batch-parsing 21641 file(s) for project 'test'...
Create new parser for project 'test'
Starting batch parsing for project 'test'...

It's just a really large dummy project:
linux kernel 2.6.29 with all files added to an empty project.
More than 21500 files, nearly 10 million lines of code (more than 6 million pure code and nearly 400 k comment and code mixed).
It uses 100% of one of my cpu's and uses more than 1.1 GB of my memory, but it seems to never end parsing.
I wait if it stops somewhen in the future, and edit this post if it happens (or if it crashes).

By the way, why are boost-headers listed here (I don't think linux-kernel uses them) ?

Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: polygon7 on October 01, 2010, 08:14:53 am
What is the meaning about this log?
Quote
** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
[New Thread 0xa9bb1b70 (LWP 30027)]

** (codeblocks:29715): CRITICAL **: smooth_draw_flat_box: assertion `width >= -1' failed
:(

I think it is related to gtk/wxgtk, but unfortunately google says nothing special about that message.
I don't remember if I see it earlier but I don't run C::B in console or in gdb very often.

I don't get it here (debian 64-bit unstable/experimental mix).
Errors like this are often caused by gtk2-changes.
Which OS are you on and what is the version of libgtk2 ?
 


It is Ubuntu 10.10 (x86) and libgtk2 version is 2.20.0-0ubuntu4 (probably, because I don't have
access to this computer at this moment - It is in my work office, and I'm on holiday (hurray! ;-D) ).
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 01, 2010, 08:59:16 am
1. support typedef of template operator overloading by blueshake, see http://forums.codeblocks.org/index.php/topic,13390.msg90193.html#msg90193 for more details.
2. fix a bug of the default parameter error in MarkItemsByAI, Now CC Search gives correct matching result.
3. improved Code-refactoring model, now it is 20X faster when verifying the search result.
4. improved template value-tooltip (typedef of template shows correct information).
5. support reparsing file after the editor modified.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 01, 2010, 05:56:38 pm
After more than 7 hours I tried to stop my experiment.
C::B used constantly about 100 % of my one cpu and memory usage increased to more than 2.2 GB.

I tried to close C::B the normal way, cpu usage reached 179 %:
After some minutes I got the message Removed test from all deps.
Memory usage was stable at 2.2 GB, cpu-usage went back to 100 % and nothing happens.
I had to kill C::B to finally close it.

"Smaller" projects like C::B are no problem.

I think I found the cause for the hang and the place, where it happens.
I will post more informations later.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: M0str3nc0 on October 01, 2010, 09:49:41 pm
Hi Everyone.

No so sure if this is the right place: I want to "improve" wxWidgets by including the newest version 2.9.1

How can I do that using the Nightly ?? Also, I carry my CodeBlocks =) =) in a USB memory because it helps me to practice C++ whenever I needed. I already downloaded wxWidgets 2.9.1 but is made from many folders and files, not just a DLL.

Any advice please ??

Thank you in advance.

M0str3nc0.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 02, 2010, 09:40:48 am
I think I found the cause for the hang and the place, where it happens.
I will post more informations later.
@jens:
Nice catch, hopefully we can fix it by your information.

Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 02, 2010, 09:42:52 am
How can I do that using the Nightly ??
??? Don't understand your problem.
to compile wxwidgets 2.9.1, search the forum and there are some discussion.
to use a portable CB, there are some info in wiki and forum. :D
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 02, 2010, 12:19:15 pm
Use real case sensitive when code completion.
If the case sensitive option checked, could be show "testFunc" only.
Code
void testFunc();
void TestFunc();
tes| // only show "testFunc" now
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 03, 2010, 12:03:16 am
It's in fact an endless loop cc runs in:
Code
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
Happens in tokenizer.cpp in CalcConditionExpression after call of ReplaceBufferForReparse in
Code
else if (!tk->m_Args.IsEmpty())
.

The internal data of m_Buffer (assigned to p in ReplaceBufferForReparse) is filled with more and more ")" characters or exactly "0x29 0x00 0x00 0x00" as it's wxChar, these characters are shown if I add a watch for p ( "')' repeats 191 times" ), but are not visible if I add a watch for m_Buffer, what seems strange, because in both cases gdb shows the value of m_Buffer.m_pchData .
Adding a watch for m_Buffer.m_pchData directly shows the same content as p.
With every loop the real length of m_Buffer grows by one (that's the cause for the great amount of memory used im my test).

I tried to strip down the sources to get a small shareable project, where that occurs, but I wasn't able to do so.
The only thing that happened was that C::B crashes with a segfault, when it tried to send the cbEVT_THREADTASK_ENDED in cbThreadPool::TaskDone without a usable backtrace, the same happens with kernel-sources from 2.6.35 kernel.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 03, 2010, 04:27:54 am
it should be a bug in Macro expansion related code. some infinite recursive macro replacement happens. like
Quote
AAA is replaced to BBB
Then BBB is replaced to AAA
...AAA is replaced to BBB
...BBB is replaced to AAA
...

Quote
With every loop the real length of m_Buffer grows by one (that's the cause for the great amount of memory used in my test).
Macro expansion is just replace some characters in the m_Buffer.

Code
xxxxxxxxxAAAA(u,v)yyyyyyyyy

For example, the above is a char Array m_Buffer, then "AAAA(u,v)" need to do a Macro expansion to some other text. So, we just do a "backward" text replace, so that, after replacement, The last replacement char was ")" in "AAAA(u,v)" (We say it as an entry point), so the text becomes:

Code
xxxNNNNNNNNNNNNNNNyyyyyyyyy

Note that "NNNNNNNNNNNN" is some macro expansion text. then the m_TokenIndex was moved backward to the beginning of the text.
if the macro expansion result text is small enough, then m_Buffer's length do not need to change.

The situation when our m_Buffer's length need to be change is that the macro expansion text is too long, so the buffer before "entry point" can not hold the new text, this way, m_Buffer's length will adjusted. like:

Code
NNNNNNNNNNNNNNNNNNNNNNyyyyyyyyy

Hope this helps to find the bug.


Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 03, 2010, 05:32:51 am
It's in fact an endless loop cc runs in:
Code
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
The replaced actual context are '_AC(x, UL)'.
ReplaceBufferForReparse() : <FROM> _AC(x,(x)<TO>_AC(x, UL)
ReplaceBufferForReparse() : <FROM> UL<TO>(x)
Happens in tokenizer.cpp in CalcConditionExpression after call of ReplaceBufferForReparse in
Code
else if (!tk->m_Args.IsEmpty())
.
It shoud be fixed by this patch.
Thanks!
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 03, 2010, 05:39:55 am
AAA is replaced to BBB
Then BBB is replaced to AAA
...AAA is replaced to BBB
...BBB is replaced to AAA
...
Yes, like this, AAA is macro, and BBB is argument. I can't write a sample too.
But i find the reason, and avoid the endless loop.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 03, 2010, 09:29:58 am
it should be a bug in Macro expansion related code. some infinite recursive macro replacement happens. like
Quote
AAA is replaced to BBB
Then BBB is replaced to AAA
...AAA is replaced to BBB
...BBB is replaced to AAA
...

Quote
With every loop the real length of m_Buffer grows by one (that's the cause for the great amount of memory used in my test).
Macro expansion is just replace some characters in the m_Buffer.

Code
xxxxxxxxxAAAA(u,v)yyyyyyyyy

For example, the above is a char Array m_Buffer, then "AAAA(u,v)" need to do a Macro expansion to some other text. So, we just do a "backward" text replace, so that, after replacement, The last replacement char was ")" in "AAAA(u,v)" (We say it as an entry point), so the text becomes:

Code
xxxNNNNNNNNNNNNNNNyyyyyyyyy

Note that "NNNNNNNNNNNN" is some macro expansion text. then the m_TokenIndex was moved backward to the beginning of the text.
if the macro expansion result text is small enough, then m_Buffer's length do not need to change.

The situation when our m_Buffer's length need to be change is that the macro expansion text is too long, so the buffer before "entry point" can not hold the new text, this way, m_Buffer's length will adjusted. like:

Code
NNNNNNNNNNNNNNNNNNNNNNyyyyyyyyy

Hope this helps to find the bug.




The cause for the growing m_Buffer is, that in every loop one of the two replacements made needed 10 bytes, but m_TokenIndex is 9, so one space is prepended.
As posted there are thousnads (after some times millions of colsing brackets visible in m_pchData but not seen by our function to display wxString's in gdb.

@ Loaden:
I test the patch and give you feedback.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 03, 2010, 10:44:53 am
@ Loaden:
I test the patch and give you feedback.

First endless-loop seems to be fixed, now we get one some lines later.

Code
ReplaceBufferForReparse() : <FROM>printf<TO>printk
ReplaceBufferForReparse() : <FROM>printk<TO>printf
tk->m_Type is printf and token is printk and vice versa.

Can this be handled in ReplaceBufferForReparse, so it only has to be done in one place ?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 03, 2010, 11:56:42 am
@ Loaden:
I test the patch and give you feedback.

First endless-loop seems to be fixed, now we get one some lines later.

Code
ReplaceBufferForReparse() : <FROM>printf<TO>printk
ReplaceBufferForReparse() : <FROM>printk<TO>printf
tk->m_Type is printf and token is printk and vice versa.

Can this be handled in ReplaceBufferForReparse, so it only has to be done in one place ?
This seems very strange! Jens, could you give me more information?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 03, 2010, 02:40:27 pm
There are several headers with #define printf printk and some others with #define printk printf.

As before I am not able to strip the sources down to a small project (the actual one uses 47 MB if it is 7zipped).

After commen ting out all #define printf's I get:
Code
ReplaceBufferForReparse() : <FROM>printk<TO>printf
ReplaceBufferForReparse() : <FROM>(             printf<TO>(args...)
ReplaceBufferForReparse() : <FROM>args...)(args)<TO>            printk(args)
ReplaceBufferForReparse() : <FROM>printk<TO>printf
ReplaceBufferForReparse() : <FROM>(             printf<TO>(args...)
ReplaceBufferForReparse() : <FROM>args...)(args)<TO>            printk(args)
ReplaceBufferForReparse() : <FROM>printk<TO>printf
ReplaceBufferForReparse() : <FROM>(             printf<TO>(args...)
endless as before, m_TokenIndex does not change if I set the brakpoint before ReplaceBufferForReparse(tk->m_Type, false); .
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 03, 2010, 04:14:27 pm
@ Loaden:
I test the patch and give you feedback.

First endless-loop seems to be fixed, now we get one some lines later.

Code
ReplaceBufferForReparse() : <FROM>printf<TO>printk
ReplaceBufferForReparse() : <FROM>printk<TO>printf
tk->m_Type is printf and token is printk and vice versa.

Can this be handled in ReplaceBufferForReparse, so it only has to be done in one place ?
Thanks! This code lead CB endless loop.
I will fix it soon.
Code
#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 03, 2010, 04:22:35 pm
Thanks! This code lead CB endless loop.
I will fix it soon.
Code
#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif

But the code above does not compile. :D
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 03, 2010, 04:30:05 pm
Thanks! This code lead CB endless loop.
I will fix it soon.
Code
#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif

But the code above does not compile. :D
But we still have to avoid all possible infinite loop. :lol:
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 03, 2010, 04:34:37 pm
Thanks! This code lead CB endless loop.
I will fix it soon.
Code
#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif

But the code above does not compile. :D

The kernel code surely does not compile also (all files added to a C::B project without running any configuration-scripts and without using makefiles), but it leads to an endless loop when parsing the sources and that should (of course) not happen, even if the code has errors.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 03, 2010, 04:40:32 pm
The kernel code surely does not compile also (all files added to a C::B project without running any configuration-scripts and without using makefiles), but it leads to an endless loop when parsing the sources and that should (of course) not happen, even if the code has errors.

But we still have to avoid all possible infinite loop. :lol:

Yes, 100% agree.
Maybe, we could limit the macro expansion level to avoid the recursive infinite loop.
Eg, once a macro definition is used to do macro expansion, it will never be used again in deeper level expansion.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 03, 2010, 04:56:10 pm
Maybe, we could limit the macro expansion level to avoid the recursive infinite loop.
Agreed! This is a good idea. :D
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 03, 2010, 05:30:52 pm
The kernel code surely does not compile also (all files added to a C::B project without running any configuration-scripts and without using makefiles), but it leads to an endless loop when parsing the sources and that should (of course) not happen, even if the code has errors.
Hi, Jens, Please trying this patch?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 03, 2010, 11:03:05 pm
The kernel code surely does not compile also (all files added to a C::B project without running any configuration-scripts and without using makefiles), but it leads to an endless loop when parsing the sources and that should (of course) not happen, even if the code has errors.
Hi, Jens, Please trying this patch?

The endless loops do no longer occur.

Parsing takes a little more than 28 minutes, that's very long, but I get more than 1 million tokens.
One problem is the amount of memory used (more than 2 GB).
And it seems parsing is not correct, I get some strange tokens in symbol browser (see the attached picture).
I am currently uploading a 7z-file, that contains the tokens tree and the outher lists from cc debug tool.
That will take some time (just ISDN here), because it's about 18 MB (105 MB tokens tree).

Another issue (not related to this), is that the symboslbrowser stays empty, if the project is opened without any shown editor.
After opening any file the symbols-browser gets filled.

Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 03, 2010, 11:51:36 pm
cc debug-info: http://apt.jenslody.de/cc/cc_debug.7z (http://apt.jenslody.de/cc/cc_debug.7z)

As written before, the test-project is generated from the (debian) linux-kernel sources, revision 2.6.29.
No build-options or defines, just all files from sources added to an empty project.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 04, 2010, 12:00:51 am
cc debug-info: http://apt.jenslody.de/cc/cc_debug.7z (http://apt.jenslody.de/cc/cc_debug.7z)

As written before, the test-project is generated from the (debian) linux-kernel sources, revision 2.6.29.
No build-options or defines, just all files from sources added to an empty project.
Thank Jens! I will look into it carefully.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 04, 2010, 02:44:18 am
Another issue (not related to this), is that the symboslbrowser stays empty, if the project is opened without any shown editor.
After opening any file the symbols-browser gets filled.
Confirmed!

EDIT: It's can confirmed only Linux, and in Windows, no have this issue.
Here:
Quote
bool Parser::Done()
{
    wxCriticalSectionLocker locker(s_ParserCritical);
    bool done =    m_UpFrontHeaders.IsEmpty()
                && m_SystemUpFrontHeaders.IsEmpty()
                && m_BatchParseFiles.IsEmpty()
                && m_PredefinedMacros.IsEmpty()
                && !m_NeedMarkFileAsLocal // When parsing end, this value be true always in Linux/GCC4.5.1!
                && m_PoolTask.empty()
                && m_Pool.Done();
    return done;
}
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 04, 2010, 03:03:21 am
answer to these questions:
Parsing takes a little more than 28 minutes, that's very long, but I get more than 1 million tokens.
1, the batch parsing time is too long:
the Linux source code you supplied contains a lot of code snippet which has grammar errors like
Code
#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif
These code will increase the loop time(at least 100 times add compare to normal code), so the performace is bad.

And it seems parsing is not correct, I get some strange tokens in symbol browser (see the attached picture).
2. parsing error, wrong tokens:
This was still due to the highly mixed pre-processor code, our CC only do a format match, like

"AAA BBB();" ---> BBB is a function.
"AAA BBB;"  ---> BBB is a variable.

Thus, CC is not so smart to detect and handle grammar errors. (the only method is do a "full pre-processor" before parsing)
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 04, 2010, 07:36:39 am
This sounds reasonable.
Nevertheless, there are tokens that contain (or even begin with) whitespace, that means these tokens are invaild, so they could be (should be ?) securely removed.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 04, 2010, 08:42:52 am
Another issue (not related to this), is that the symboslbrowser stays empty, if the project is opened without any shown editor.
After opening any file the symbols-browser gets filled.
Confirmed!

EDIT: It's can confirmed only Linux, and in Windows, no have this issue.
Here:
Quote
bool Parser::Done()
{
    wxCriticalSectionLocker locker(s_ParserCritical);
    bool done =    m_UpFrontHeaders.IsEmpty()
                && m_SystemUpFrontHeaders.IsEmpty()
                && m_BatchParseFiles.IsEmpty()
                && m_PredefinedMacros.IsEmpty()
                && !m_NeedMarkFileAsLocal // When parsing end, this value be true always in Linux/GCC4.5.1!
                && m_PoolTask.empty()
                && m_Pool.Done();
    return done;
}
There seem to be two instances of NativeParser with two different variables for m_Parser, one parser returns true for Done() and the other one returns false.
The one that returns true is called in CodeCompletion::OnParserEnd and the other one is the one that catches the PARSER_END event in NativeParser::OnParserEnd .
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on October 04, 2010, 09:53:47 am
please have a look at this. It seems a little regression got introduced : http://forums.codeblocks.org/index.php/topic,13413.msg90463.html#msg90463
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 04, 2010, 10:11:26 am
please have a look at this. It seems a little regression got introduced : http://forums.codeblocks.org/index.php/topic,13413.msg90463.html#msg90463
OK, I will look into it, and hopefully fixed soon.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 04, 2010, 04:19:28 pm
There seem to be two instances of NativeParser with two different variables for m_Parser, one parser returns true for Done() and the other one returns false.
The one that returns true is called in CodeCompletion::OnParserEnd and the other one is the one that catches the PARSER_END event in NativeParser::OnParserEnd .
Fixed in r6665, thanks!
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 05, 2010, 08:38:35 am
Nevertheless, there are tokens that contain (or even begin with) whitespace, that means these tokens are invaild, so they could be (should be ?) securely removed.
I found a reason, in file "/home/jens/kernel-tmp/linux-source-2.6.29/sound/aoa/codecs/tas.c", there has a struct named tas:
Code
struct tas {
    struct aoa_codec    codec;
    struct i2c_client    *i2c;
    u32            mute_l:1, mute_r:1 ,
                controls_created:1 ,
                drc_enabled:1,
                hw_enabled:1;
    u8            cached_volume_l, cached_volume_r;
    u8            mixer_l[3], mixer_r[3];
    u8            bass, treble;
    u8            acr;
    int            drc_range;
    /* protects hardware access against concurrency from
     * userspace when hitting controls and during
     * codec init/suspend/resume */
    struct mutex        mtx;
};
and in "/home/jens/kernel-tmp/linux-source-2.6.29/arch/cris/include/arch-v32/arch/system.h"
Here a macro define, like this:
Quote
preprocessor #define tas(ptr) (xchg((ptr),1))    [186,42]
So, we will get a error parse result:
Quote
/home/jens/kernel-tmp/linux-source-2.6.29/sound/aoa/codecs/tas.c
preprocessor #define PFX DEVNAME ": "    [1,77]
class class (xchg((ptr),1)) {...}    [83,83]
variable u32 (xchg((ptr),1))::mute_l    [86,0]
variable u32 (xchg((ptr),1))::mute_r    [86,0]
variable u32 (xchg((ptr),1))::controls_created    [87,0]
variable u32 (xchg((ptr),1))::drc_enabled    [88,0]
variable u32 (xchg((ptr),1))::hw_enabled    [89,0]
variable u32u8 (xchg((ptr),1))::cached_volume_l    [90,0]
variable u32u8 (xchg((ptr),1))::cached_volume_r    [90,0]
variable u8 (xchg((ptr),1))::mixer_l    [91,0]
variable u8 (xchg((ptr),1))::mixer_r    [91,0]
variable u8 (xchg((ptr),1))::bass    [92,0]
variable u8 (xchg((ptr),1))::treble    [92,0]
variable u8 (xchg((ptr),1))::acr    [93,0]
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 05, 2010, 08:41:56 am
Here is a test code snippet:
Code
#define tas(ptr) (xchg((ptr),1))

struct tas {
    struct aoa_codec    codec;
    struct i2c_client    *i2c;
    u32            mute_l:1, mute_r:1 ,
                controls_created:1 ,
                drc_enabled:1,
                hw_enabled:1;
    u8            cached_volume_l, cached_volume_r;
    u8            mixer_l[3], mixer_r[3];
    u8            bass, treble;
    u8            acr;
    int            drc_range;
    /* protects hardware access against concurrency from
     * userspace when hitting controls and during
     * codec init/suspend/resume */
    struct mutex        mtx;
};
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 05, 2010, 09:19:06 am
Nevertheless, there are tokens that contain (or even begin with) whitespace, that means these tokens are invaild, so they could be (should be ?) securely removed.
This issue shoud be fixed in r6670. :D
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 05, 2010, 07:02:26 pm
Next endless loop, this time with linux kernel-sources 2.6.35.
It happens in parserthread.cpp:2226 ( m_Tokenizer.ReplaceBufferForReparse(m_Tokenizer.GetActualContextForMacro(tk)); ).

Snippet of debug-output parserthread and tokenizer:

Code
DoParse() : Loop:m_Str='', token='dma_addr_t'
ReadParentheses(): (* map_single), line=355
HandleMacro() : Adding token 'dma_addr_t' (peek='(* map_single)')
DoAddToken() : Found token (parent).
GetActualTokenType() : Searching within m_Str=''
GetActualTokenType() : Compensated m_Str=''
GetActualTokenType() : Returning ''
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'dma_addr_t' (490410), type '', actual ''. Parent is esp_driver_ops (490406)
The replaced actual context are 'dma_addr_t'.
ReplaceBufferForReparse() : <FROM>dma_addr_t<TO>dma_addr_t
DoParse() : Loop:m_Str='', token='dma_addr_t'
ReadParentheses(): (* map_single), line=355
HandleMacro() : Adding token 'dma_addr_t' (peek='(* map_single)')
DoAddToken() : Found token (parent).
GetActualTokenType() : Searching within m_Str=''
GetActualTokenType() : Compensated m_Str=''
GetActualTokenType() : Returning ''
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'dma_addr_t' (490410), type '', actual ''. Parent is esp_driver_ops (490406)
The replaced actual context are 'dma_addr_t'.
ReplaceBufferForReparse() : <FROM>dma_addr_t<TO>dma_addr_t
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 06, 2010, 06:17:02 am
Next endless loop, this time with linux kernel-sources 2.6.35.
Fixed in r6671.
Thanks!
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 06, 2010, 09:41:07 am
Next endless loop, this time with linux kernel-sources 2.6.35.
Fixed in r6671.
Thanks!

Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Thank you !

And also the annoying issue, that the symbols-browser is initially empty if no file is opened, seems to be fixed !
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 06, 2010, 10:10:36 am
Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Hi, jens, Did you remember how long will the batch parse take to parse the whole Linux source, when you did this last time(If I remember correct, maybe one year ago)?



Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 06, 2010, 10:14:50 am
And also the annoying issue, that the symbols-browser is initially empty if no file is opened, seems to be fixed !
Yes, This issue fixed in r6665.
Quote
* cc_branch: fix parser init error, lead class browser can not correctly refreshed

-------------------------------
M : /branches/codecompletion_refactoring/src/plugins/codecompletion/nativeparser.cpp 
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 06, 2010, 10:19:59 am
Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).
Hi, Jens, About the batch parsing time, could you shut off the debug log trace, and trying again?
Here:
Quote
#define CC_TOKENIZER_DEBUG_OUTPUT 0
#define CC_PARSERTHREAD_DEBUG_OUTPUT 0
...
If use debug log, will lead batch parsing as long time.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 06, 2010, 10:21:53 am
Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Hi, jens, Did you remember how long will the batch parse take to parse the whole Linux source, when you did this last time(If I remember correct, maybe one year ago)?

No, and it would not give any answers about incfreasing or decreasing parse time, because this time I parsed 2.6.35 kernel sources, the last time it was 2.6.29 with only 21k files and a little more than a million tokens, but I will test with current trunk and give you feedback.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 06, 2010, 10:25:08 am
Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).
Hi, Jens, About the batch parsing time, could you shut off the debug log trace, and trying again?
Here:
Quote
#define CC_TOKENIZER_DEBUG_OUTPUT 0
#define CC_PARSERTHREAD_DEBUG_OUTPUT 0
...
If use debug log, will lead batch parsing as long time.

I can do, but it will not make such a big difference I guess, because I have an additional static variable inside the ifdef to switch on debug output while running debugger (by changing the value from false to true).
But I will test anyway.
When running batch output all the time it would last much longer.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 06, 2010, 10:28:33 am
Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Hi, jens, Did you remember how long will the batch parse take to parse the whole Linux source, when you did this last time(If I remember correct, maybe one year ago)?

No, and it would not give any answers about incfreasing or decreasing parse time, because this time I parsed 2.6.35 kernel sources, the last time it was 2.6.29 with only 21k files and a little more than a million tokens, but I will test with current trunk and give you feedback.
There may be a large number of duplicate tokens, tokens will not be in direct proportion with the time.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: polygon7 on October 06, 2010, 02:47:53 pm
Hi,
I'm testing C::B CC_branch on another big project - OpenOffice and after ~20 minutes CC parser is hanging on:

Code
InitTokenizer() : m_Filename='/usr/include/bits/time.h', m_FileSize=2866.
Parse() : Parsing '/usr/include/bits/time.h'
DoParse() : Loop:m_Str='', token='#'
DoParse() : Loop:m_Str='', token='#'
DoAddToken() : Created token='_STRUCT_TIMEVAL', file_idx=12732, line=70, ticket=
GetActualTokenType() : Searching within m_Str='1'
GetActualTokenType() : Compensated m_Str='1'
GetActualTokenType() : Found '1'
DoAddToken() : Prepending ''
DoParse() : Loop:m_Str='', token='#'
HandleIncludes() : Found include file 'bits/types.h'
DoParse() : Loop:m_Str='', token='struct'
HandleClass() : Found class 'timeval'
DoAddToken() : Created token='timeval', file_idx=12732, line=75, ticket=
GetActualTokenType() : Searching within m_Str=''
GetActualTokenType() : Compensated m_Str=''
GetActualTokenType() : Returning ''
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'timeval' (353789), type '', actual ''. Parent is  (-1)
DoParse() : Loop:m_Str='', token='__time_t'
DoParse() : Loop:m_Str='__time_t ', token='tv_sec'
DoAddToken() : Created token='tv_sec', file_idx=12732, line=77, ticket=
GetActualTokenType() : Searching within m_Str='__time_t'
GetActualTokenType() : Compensated m_Str='__time_t'
GetActualTokenType() : Found '__time_t'
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'tv_sec' (353790), type '__time_t', actual '__time_t'. Parent is timeval (353789)
DoParse() : Loop:m_Str='__time_t', token=';'
DoParse() : Loop:m_Str='', token='__suseconds_t'
DoParse() : Loop:m_Str='__suseconds_t ', token='tv_usec'
DoAddToken() : Created token='tv_usec', file_idx=12732, line=78, ticket=
GetActualTokenType() : Searching within m_Str='__suseconds_t'
GetActualTokenType() : Compensated m_Str='__suseconds_t'
GetActualTokenType() : Found '__suseconds_t'
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'tv_usec' (353791), type '__suseconds_t', actual '__suseconds_t'. Parent is timeval (353789)
DoParse() : Loop:m_Str='__suseconds_t', token=';'
DoParse() : Loop:m_Str='', token='}'
InitTokenizer() : m_Filename='/usr/include/bits/dirent.h', m_FileSize=1609.
Parse() : Parsing '/usr/include/bits/dirent.h'
DoParse() : Loop:m_Str='', token='struct'
HandleClass() : Found class 'dirent'
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...

Code
Tasks: 143 total,   1 running, 142 sleeping,   0 stopped,   0 zombie
Cpu(s): 60.8%us,  3.0%sy,  0.0%ni, 36.0%id,  0.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2062772k total,  1969520k used,    93252k free,   145608k buffers
Swap:  2104476k total,     9292k used,  2095184k free,   664664k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                            
15947 1234   20   0  824m 573m  34m S  102 28.5  19:55.35 codeblocks


// EDIT: C::B CC branch rev 6671
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 06, 2010, 02:53:19 pm
@polygon7 (http://forums.codeblocks.org/index.php?action=profile;u=270)
from the log, I can't find the hint where there is a infinite loop, it seem the token line becomes bigger and bigger.

But there should be some thing wrong in Parserthread, we will check it.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 06, 2010, 03:03:29 pm
Without my changes (static variable for deug and some if-clauses in define) I can no longer use C::B to parse the 2.6.35 kernel.
If I run it from commandline, it eats up all my memory (up to ~4GB) and after some tome it crashes with an X-window error (resource temproary unavailable), if I run it through debugger it crashes with a segfault in wxPostEVent, if it tries to send a TaskDone-event.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 06, 2010, 03:05:30 pm
Here are the results of measuring and the last lines of debug-output before it crashes:

Code
trunk:
[...]
33115 files loaded
Done loading project in 84084ms
[...]
Parsing stage done (33919 total parsed files, 1193078 tokens in 1 minute(s), 59.116 seconds).


"Pure" cc:
[...]
33115 files loaded
Done loading project in 82294ms
[...]

DoAddToken() : Created token='__xl', file_idx=761, line=27, ticket=
GetActualTokenType() : Searching within m_Str='"r0"'
GetActualTokenType() : Compensated m_Str='"r0"'
GetActualTokenType() : Found ''
DoAddToken() : Prepending ''
DoParse() : Loop:m_Str='', token='#'
wxString Tokenizer::ReadToEOL(bool, bool) : line=28, CurrentChar='
', PreviousChar='"', NextChar='#', nestBrace(1)
ReadToEOL(): (END) We are now at line 28, CurrentChar='
', PreviousChar='"', NextChar='#'
ReadToEOL():
DoAddToken() : Created token='__xh', file_idx=761, line=28, ticket=
GetActualTokenType() : Searching within m_Str='"r1"'
GetActualTokenType() : Compensated m_Str='"r1"'
GetActualTokenType() : Found ''
DoAddToken() : Prepending ''
HandleConditionPreprocessor() : #endif at line = 29
bool Tokenizer::SkipToEOL(bool) : line=29, CurrentChar='
', PreviousChar='f', NextChar='
', nestBrace(0)
SkipToEOL(): (END) We are now at line 29, CurrentChar='
', PreviousChar='f', NextChar='
'
DoParse() : Loop:m_Str='', token='#'
ReadParentheses(): (n, base), line=31
wxString Tokenizer::ReadToEOL(bool, bool) : line=31, CurrentChar='      ', PreviousChar=')', NextChar=' ', nestBrace(1)
ReadToEOL(): (END) We are now at line 47, CurrentChar='
', PreviousChar=')', NextChar='
'
ReadToEOL():                                    ({                                                                      register unsigned int __base asm("r4") = base;          register unsigned long long __n asm("r0") = n;          register unsigned long long __res asm("r2");                    register unsigned int __rem asm(__xh);                   asm(    __asmeq("%0", __xh)                                             __asmeq("%1", "r2")                                             __asmeq("%2", "r0")                                             __asmeq("%3", "r4")                                             "bl     __do_div64"                                              : "=r" (__rem), "=r" (__res)                                    : "r" (__n), "r" (__base)                                       : "ip", "lr", "cc");                                    n = __res;                                                      __rem;  })
DoAddToken() : Created token='__do_div_asm', file_idx=761, line=31, ticket=
GetActualTokenType() : Searching within m_Str='                                 ({                                                                      register unsigned int __base asm("r4") = base;          register unsigned long long __n asm("r0") = n;          register unsigned long long __res asm("r2");    register unsigned int __rem asm(__xh);                   asm(    __asmeq("%0", __xh)                                             __asmeq("%1", "r2")                                             __asmeq("%2", "r0")                                             __asmeq("%3", "r4")                                     "bl      __do_div64"                                             : "=r" (__rem), "=r" (__res)                                    : "r" (__n), "r" (__base)                                       : "ip", "lr", "cc");                                    n = __res;                                                      __rem;                                                   })'
GetActualTokenType() : Compensated m_Str='                                      ({                                                                      register unsigned int __base asm("r4") = base;          register unsigned long long __n asm("r0") = n;          register unsigned long long __res asm("r2");    register unsigned int __rem asm(__xh);                   asm(    __asmeq("%0", __xh)                                             __asmeq("%1", "r2")                                             __asmeq("%2", "r0")                                             __asmeq("%3", "r4")                                     "bl      __do_div64"                                             :"=r" (__rem), "=r" (__res)                                     :"r" (__n), "r" (__base)                                        :"ip", "lr", "cc");                                     n = __res;                                                      __rem;                                                   })'
GetActualTokenType() : Found ''
DoAddToken() : Prepending ''
HandleConditionPreprocessor() : #if at line = 49
bool Tokenizer::SkipToEOL(bool) : line=49, CurrentChar=' ', PreviousChar='f', NextChar='_', nestBrace(0)
SkipToEOL(): (END) We are now at line 49, CurrentChar='
', PreviousChar='4', NextChar='
'
CalcConditionExpression() : exp.GetStatus() : 1, exp.GetResult() : 0
HandleConditionPreprocessor() : #elif at line = 61
bool Tokenizer::SkipToEOL(bool) : line=61, CurrentChar=' ', PreviousChar='f', NextChar='_', nestBrace(0)
SkipToEOL(): (END) We are now at line 61, CurrentChar='
', PreviousChar='4', NextChar='
'
CalcConditionExpression() : exp.GetStatus() : 1, exp.GetResult() : 1
DoParse() : Loop:m_Str='', token='#'
HandleIncludes() : Found include file 'asm/bug.h'
DoParse() : Loop:m_Str='', token='#'
ReadParentheses(): (n, base), line=73
wxString Tokenizer::ReadToEOL(bool, bool) : line=73, CurrentChar='      ', PreviousChar=')', NextChar=' ', nestBrace(1)
ReadToEOL(): (END) We are now at line 211, CurrentChar='
', PreviousChar=')', NextChar='
'
 
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 07, 2010, 01:50:43 am
thanks jens for the test.
From the "pure CC" log file before crash, I can't find any thing wrong, it seems the log was correct (parserthread works fine before crash).
So, the problem should happened before the crash.

@loaden, Maybe, we could add an option to "disable the macro expansion", then see if it still hangs or crash in this case.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 07, 2010, 07:20:09 am
thanks jens for the test.
From the "pure CC" log file before crash, I can't find any thing wrong, it seems the log was correct (parserthread works fine before crash).
So, the problem should happened before the crash.

@loaden, Maybe, we could add an option to "disable the macro expansion", then see if it still hangs or crash in this case.

My guess:
the crash happens, because the applications runs out of memory.

If I run it from commandline, it eats up all my memory (up to ~4GB) and after some tome it crashes with an X-window error (resource temproary unavailable), if I run it through debugger it crashes with a segfault in wxPostEVent, if it tries to send a TaskDone-event.

The crash occurs if I switch to the running C::B, if it's visible and I run it from inside C::B (through gdb) it crashes in wxPostEvent.
Maybe the size of the used buffers should also be traced.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 07, 2010, 07:53:21 am
Maybe the size of the used buffers should also be traced.
Yes, I'll checking carefully.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 07, 2010, 03:02:02 pm
Hi, Jens, could you trying r6676?
I am testing in XP, it seems solved.
Quote
Project 'linux' parsing stage done (27214 total parsed files, 1342720 tokens in 17 minute(s), 37.094 seconds).
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 07, 2010, 03:29:35 pm
If set the option "Editor > CC > C/C++ Parser > Parse complex macros" is no checked, will lead the time is less.
Quote
Project 'linux' parsing stage done (27214 total parsed files, 1258851 tokens in 8 minute(s), 54.969 seconds).
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 07, 2010, 11:17:22 pm
linux-kernel 2.6.29
with complex macros:
Code
Project 'test' parsing stage done (21827 total parsed files, 1011919 tokens in 22 minute(s), 2.405 seconds).
without complex macros:
Code
Project 'test' parsing stage done (21827 total parsed files, 951347 tokens in 3 minute(s), 58.764 seconds).

linux-kernel 2.6.35
with complex macros:
Code
Project 'test' parsing stage done (30453 total parsed files, 1343328 tokens in 41 minute(s), 5.657 seconds).
without complex macros and without parsing preprocessor:
Code
Project 'test' parsing stage done (30453 total parsed files, 1259360 tokens in 4 minute(s), 50.464 seconds).

The second time (parsing without) was done immediately after closing the project, so that most of the files are still in the systems hdd-buffer.

Parsing the linux-kernel 2.6.35 only works with debug-output enebaled and redirected to a file (C::B uses up to 2.5 GB of memory and did not release it at project unload).
Using from within C::B or diretcly from console, either crashes C::B or freezes the whole system.

By the way,  the following patch is needed to make C::B compile without a warning if CC_XXX_DEBUG_OUTPUT is set to 2.
I did not change all places (should all be checked I think), only the places which lead to a compiler warning:
Code
Index: src/plugins/codecompletion/parser/token.cpp
===================================================================
--- src/plugins/codecompletion/parser/token.cpp (Revision 6677)
+++ src/plugins/codecompletion/parser/token.cpp (Arbeitskopie)
@@ -1027,10 +1027,10 @@
         }
     }
 
-#if CC_TOKEN_DEBUG_OUTPUT
-    TRACE(_T("RecalcInheritanceChain() : First iteration took : %ld ms"), sw.Time());
-    sw.Start();
-#endif
+//#if CC_TOKEN_DEBUG_OUTPUT
+//    TRACE(_T("RecalcInheritanceChain() : First iteration took : %ld ms"), sw.Time());
+//    sw.Start();
+//#endif
 
     // recalc
     TokenIdxSet result;
@@ -1057,16 +1057,20 @@
         {
             Token* anc_token = at(*it);
             if (anc_token)
+            {
                 TRACE(_T("RecalcInheritanceChain() :  + %s"), anc_token->m_Name.wx_str());
+            }
             else
+            {
                 TRACE(_T("RecalcInheritanceChain() :  + NULL?!"));
+            }
         }
     }
 #endif
 
-#if CC_TOKEN_DEBUG_OUTPUT
-    TRACE(_T("RecalcInheritanceChain() : Second iteration took : %ld ms"), sw.Time());
-#endif
+//#if CC_TOKEN_DEBUG_OUTPUT
+//    TRACE(_T("RecalcInheritanceChain() : Second iteration took : %ld ms"), sw.Time());
+//#endif
 
     TRACE(_T("RecalcInheritanceChain() : Full inheritance calculated."));
 }
@@ -1215,9 +1219,13 @@
             {
                 Token* anc_token = at(*it);
                 if (anc_token)
+                {
                     TRACE(_T("RecalcData() :  + %s"), anc_token->m_Name.wx_str());
+                }
                 else
+                {
                     TRACE(_T("RecalcData() :  + NULL?!"));
+                }
             }
         }
 #endif
Index: src/plugins/codecompletion/parser/parserthread.cpp
===================================================================
--- src/plugins/codecompletion/parser/parserthread.cpp (Revision 6677)
+++ src/plugins/codecompletion/parser/parserthread.cpp (Arbeitskopie)
@@ -1197,8 +1197,10 @@
 
         newToken = new(std::nothrow) Token(newname, m_FileIdx, line, ++m_pTokensTree->m_TokenTicketCount);
         if (newToken)
+        {
             TRACE(_T("DoAddToken() : Created token='%s', file_idx=%d, line=%d, ticket="), newname.wx_str(),
                   m_FileIdx, line, m_pTokensTree->m_TokenTicketCount);
+        }
         else
         {
             --m_pTokensTree->m_TokenTicketCount;

The commented out part (with the wxStopWatch) should probably be corrected instead of just commented out (at least if it still makes sense to measure the times).
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 08, 2010, 07:12:06 am
By the way,  the following patch is needed to make C::B compile without a warning if CC_XXX_DEBUG_OUTPUT is set to 2.
I did not change all places (should all be checked I think), only the places which lead to a compiler warning:
Fixed in r6678.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 08, 2010, 07:44:56 am
The second time (parsing without) was done immediately after closing the project, so that most of the files are still in the systems hdd-buffer.
No, because the ParserThread instance of these files does not created.

Parsing the linux-kernel 2.6.35 only works with debug-output enebaled and redirected to a file (C::B uses up to 2.5 GB of memory and did not release it at project unload).
Using from within C::B or diretcly from console, either crashes C::B or freezes the whole system.
I can not reproduce in the Windows system, you can help find which file is causing the problem?
Thanks!
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Borr on October 08, 2010, 09:23:06 am
CB_CC_BRANCH_r6675 can't parce ole2.h with MinGW

Code
#include <ole2.h>
...
VARIANT param;
VariantInit(&param);
param./*Ctrl-Space show only dblVal*/
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 08, 2010, 11:27:58 am
CB_CC_BRANCH_r6675 can't parce ole2.h with MinGW

Code
#include <ole2.h>
...
VARIANT param;
VariantInit(&param);
param./*Ctrl-Space show only dblVal*/

Sorry, this is too complex, my personal opinion, not yet plan to support it.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 08, 2010, 12:05:27 pm
CB_CC_BRANCH_r6675 can't parce ole2.h with MinGW

Code
#include <ole2.h>
...
VARIANT param;
VariantInit(&param);
param./*Ctrl-Space show only dblVal*/

Sorry, this is too complex, my personal opinion, not yet plan to support it.

there are some unnamed union in the declaration of VARIANT. I'm trying to figure it out. but the latest cc_branch get crashed when I try to view ole2.h.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 08, 2010, 12:09:33 pm
The second time (parsing without) was done immediately after closing the project, so that most of the files are still in the systems hdd-buffer.
No, because the ParserThread instance of these files does not created.

...I'm confused about your discussion. Jens says that the second batch parse takes much less time because of the system's cache for source files.

Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 08, 2010, 01:31:03 pm
The second time (parsing without) was done immediately after closing the project, so that most of the files are still in the systems hdd-buffer.
No, because the ParserThread instance of these files does not created.

...I'm confused about your discussion. Jens says that the second batch parse takes much less time because of the system's cache for source files.
I see, thanks!
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 09, 2010, 06:39:43 am
@killerbot
Could you publish a CODECOMPLETION BRANCH nightly build in this weekend?
We add some feature, and fixed some bugs of reported in forum.
Thanks a lot! :D

EDIT: Please waiting, found a new issue, we need a whole day for test. :(
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 09, 2010, 06:52:06 am
@MortenMacFly
Maybe we need merged with trunk (trunk to cc_branch) ?
We need your help. :)
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 09, 2010, 12:08:54 pm
@killerbot
Could you publish a CODECOMPLETION BRANCH nightly build in this weekend?
We add some feature, and fixed some bugs of reported in forum.
Thanks a lot! :D

EDIT: Please waiting, found a new issue, we need a whole day for test. :(
Fixed in r6687, so, I think we are ready now. :D
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 10, 2010, 02:21:00 pm
Hi,
I'm testing C::B CC_branch on another big project - OpenOffice and after ~20 minutes CC parser is hanging on:

Code
InitTokenizer() : m_Filename='/usr/include/bits/time.h', m_FileSize=2866.
Parse() : Parsing '/usr/include/bits/time.h'
DoParse() : Loop:m_Str='', token='#'
DoParse() : Loop:m_Str='', token='#'
DoAddToken() : Created token='_STRUCT_TIMEVAL', file_idx=12732, line=70, ticket=
GetActualTokenType() : Searching within m_Str='1'
GetActualTokenType() : Compensated m_Str='1'
GetActualTokenType() : Found '1'
DoAddToken() : Prepending ''
DoParse() : Loop:m_Str='', token='#'
HandleIncludes() : Found include file 'bits/types.h'
DoParse() : Loop:m_Str='', token='struct'
HandleClass() : Found class 'timeval'
DoAddToken() : Created token='timeval', file_idx=12732, line=75, ticket=
GetActualTokenType() : Searching within m_Str=''
GetActualTokenType() : Compensated m_Str=''
GetActualTokenType() : Returning ''
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'timeval' (353789), type '', actual ''. Parent is  (-1)
DoParse() : Loop:m_Str='', token='__time_t'
DoParse() : Loop:m_Str='__time_t ', token='tv_sec'
DoAddToken() : Created token='tv_sec', file_idx=12732, line=77, ticket=
GetActualTokenType() : Searching within m_Str='__time_t'
GetActualTokenType() : Compensated m_Str='__time_t'
GetActualTokenType() : Found '__time_t'
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'tv_sec' (353790), type '__time_t', actual '__time_t'. Parent is timeval (353789)
DoParse() : Loop:m_Str='__time_t', token=';'
DoParse() : Loop:m_Str='', token='__suseconds_t'
DoParse() : Loop:m_Str='__suseconds_t ', token='tv_usec'
DoAddToken() : Created token='tv_usec', file_idx=12732, line=78, ticket=
GetActualTokenType() : Searching within m_Str='__suseconds_t'
GetActualTokenType() : Compensated m_Str='__suseconds_t'
GetActualTokenType() : Found '__suseconds_t'
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'tv_usec' (353791), type '__suseconds_t', actual '__suseconds_t'. Parent is timeval (353789)
DoParse() : Loop:m_Str='__suseconds_t', token=';'
DoParse() : Loop:m_Str='', token='}'
InitTokenizer() : m_Filename='/usr/include/bits/dirent.h', m_FileSize=1609.
Parse() : Parsing '/usr/include/bits/dirent.h'
DoParse() : Loop:m_Str='', token='struct'
HandleClass() : Found class 'dirent'
C++ Parser is still parsing files...
C++ Parser is still parsing files...
C++ Parser is still parsing files...

Code
Tasks: 143 total,   1 running, 142 sleeping,   0 stopped,   0 zombie
Cpu(s): 60.8%us,  3.0%sy,  0.0%ni, 36.0%id,  0.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2062772k total,  1969520k used,    93252k free,   145608k buffers
Swap:  2104476k total,     9292k used,  2095184k free,   664664k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                            
15947 1234   20   0  824m 573m  34m S  102 28.5  19:55.35 codeblocks


// EDIT: C::B CC branch rev 6671
Could you trying r6690?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: polygon7 on October 11, 2010, 04:41:30 pm
Could you trying r6690?

I have tested it with rev. 6702 and CodeCompletion works ("Parse complex macros" is enabled):
Code
Loading project files...
23587 files loaded
Done loading project in 66327ms
Project's base path: /home/lukaszg/projekty/openoffice/
Project's common toplevel path: /home/lukaszg/projekty/openoffice/
[New Thread 0xa8e53b70 (LWP 21390)]
Caching GCC dir: /usr/include/c++/4.5.1
Caching GCC dir: /usr/include/c++/4.5.1/i686-pc-linux-gnu
Caching GCC dir: /usr/include/c++/4.5.1/backward
Caching GCC dir: /usr/local/include
Caching GCC dir: /usr/lib/gcc/i686-pc-linux-gnu/4.5.1/include
Caching GCC dir: /usr/lib/gcc/i686-pc-linux-gnu/4.5.1/include-fixed
Caching GCC dir: /usr/include
Passing list of files to batch-parser.
Header to parse up-front: '/usr/include/c++/4.5.1/cstddef'
Header to parse up-front: '/usr/include/boost/config.hpp'
Header to parse up-front: '/usr/include/boost/filesystem/config.hpp'
Header to parse up-front: '/home/lukaszg/projekty/openoffice/embedserv/source/inc/stdafx.h'
Add up-front parsing 4 file(s) for project 'openoffice'...
Add batch-parsing 12303 file(s) for project 'openoffice'...
Create new parser for project 'openoffice'
EnvVars: Obtained 'default' as active envvar set from config.
EnvVars: Set 'default' will not be applied (already active).
Starting batch parsing for project 'openoffice'...
[New Thread 0xa8652b70 (LWP 21395)]

Project 'openoffice' parsing stage done (13394 total parsed files, 403357 tokens in 1 minute(s), 46.055 seconds).

... but there is a problem when I'm trying to write:
Code
#include <
or
Code
#include "
to include some file then C::B have ~2-7 sec. "hang" (project type or size doesn't matter).

My default (global) compiler search directories are:
Code
/usr/include
/usr/include/boost
/usr/include/libxml2
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on October 11, 2010, 05:07:23 pm
yes, this delay is already present for so time now, I can confirm ;-)
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 11, 2010, 05:12:35 pm
yes, this delay is already present for so time now, I can confirm ;-)
Because there has too many system headers, and seems need a fast IO system. :)
And now, I have not good idea. :(
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: polygon7 on October 11, 2010, 05:33:25 pm
yes, this delay is already present for so time now, I can confirm ;-)
Because there has too many system headers, and seems need a fast IO system. :)
And now, I have not good idea. :(

I don't know - maybe C::B could use some kind of "memory cache" for global / project include file names
- on C::B start one thread could insert global includes file names into some std::map<filename, path>(or maybe other container),
and when user open some project another thread could insert per project file names into same container?

//Edit: And when user write:
Code
#include "
C::B will use that "cache"?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 11, 2010, 05:38:25 pm
yes, this delay is already present for so time now, I can confirm ;-)
Because there has too many system headers, and seems need a fast IO system. :)
And now, I have not good idea. :(

I don't know - maybe C::B could use some kind of "memory cache" for global / project include file names
- on C::B start one thread could insert global includes file names into some std::map<filename, path>(or maybe other container),
and when user open some project another thread could insert per project file names into same container?

//Edit: And when user write:
Code
#include "
C::B will use that "cache"?
Yes, use std::map<wxString, std::list<wxString> > to save all system headers.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 11, 2010, 08:39:34 pm
linux-kernel 2.6.29:
Code
Project 'test' parsing stage done (21821 total parsed files, 910203 tokens in 3 minute(s), 11.931 seconds).

linux-kernel 2.6.35:
Code
Project 'test' parsing stage done (30447 total parsed files, 1313354 tokens in 6 minute(s), 54.779 seconds).

Good work !
 I am just compiling it on my server, to update the cc-repo.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 12, 2010, 01:56:23 am
I am just compiling it on my server, to update the cc-repo.
:D Thanks!
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 12, 2010, 08:55:28 am
yes, this delay is already present for so time now, I can confirm ;-)
Hi, killerbot, Could you have a trying r6707.
And post the debug log in here?
Thanks!
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: polygon7 on October 12, 2010, 11:14:19 am
yes, this delay is already present for so time now, I can confirm ;-)
Hi, killerbot, Could you have a trying r6707.
And post the debug log in here?
Thanks!
Revision 6707 works really fast with #include's :D

// Edit:
include with "
Code
Get include file count is 2, use time is 1
include with <
Code
Get include file count is 31006, use time is 57
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 12, 2010, 12:20:12 pm
yes, this delay is already present for so time now, I can confirm ;-)
Hi, killerbot, Could you have a trying r6707.
And post the debug log in here?
Thanks!
Revision 6707 works really fast with #include's :D

// Edit:
include with "
Code
Get include file count is 2, use time is 1
include with <
Code
Get include file count is 31006, use time is 57

About the 2~6 sec delay: I think it is wxScintilla issue, not CC. :(
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on October 12, 2010, 02:34:48 pm
#include " --> worked very fast, but several includes missing
#include <  --> still rather slow [and at the start I always have a lot of entries like this : /32/32/32/32/32/32/32....]
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 12, 2010, 04:00:57 pm
#include " --> worked very fast, but several includes missing
#include <  --> still rather slow [and at the start I always have a lot of entries like this : /32/32/32/32/32/32/32....]
All code in here:
Quote
// Do the code completion when we enter:
// #include "| or #include <|
void CodeCompletion::CodeCompleteIncludes()
{
    wxStopWatch sw;

    if (!IsAttached() || !m_InitDone)
        return;

    cbEditor* ed = Manager::Get()->GetEditorManager()->GetBuiltinActiveEditor();
    if (!ed)
        return;

    const wxString curFile(ed->GetFilename());
    const wxString curPath(wxFileName(curFile).GetPath());
    wxArrayString buildTargets;

    cbProject* project = m_NativeParser.GetProjectByParser(&m_NativeParser.GetParser());
    ProjectFile* pf = project ? project->GetFileByFilename(curFile, false) : 0;
    if (pf)
        buildTargets = pf->buildTargets;

    FileType ft = FileTypeOf(ed->GetShortName());
    if ( ft != ftHeader && ft != ftSource) // only parse source/header files
        return;

    int pos = ed->GetControl()->GetCurrentPos();
    int lineStartPos = ed->GetControl()->PositionFromLine(ed->GetControl()->GetCurrentLine());
    wxString line = ed->GetControl()->GetLine(ed->GetControl()->GetCurrentLine());
    line.Trim();
    if (line.IsEmpty() || !TestIncludeLine(line))
        return;

    bool useSystemHeaders = false;
    int keyPos = line.Find(_T('"'));
    if (keyPos == wxNOT_FOUND)
    {
        useSystemHeaders = true;
        keyPos = line.Find(_T('<'));
    }
    if (keyPos == wxNOT_FOUND || keyPos > pos - lineStartPos)
        return;
    ++keyPos;

    // now, we are after the quote prompt
    wxString filename = line.SubString(keyPos, pos - lineStartPos - keyPos);
    filename.Replace(_T("\\"), _T("/"), true);

    // fill a list of matching files
    StringSet files;

    // #include <|
    if (m_CCSystemHeaderFiles && useSystemHeaders)
    {
        wxCriticalSectionLocker locker(s_HeadersCriticalSection);
        wxArrayString& incDirs = GetSystemIncludeDirs(&m_NativeParser.GetParser(),
                                                      project ? project->GetModified() : true);
        for (size_t i = 0; i < incDirs.GetCount(); ++i)
        {
            SystemHeadersMap::iterator it = m_SystemHeadersMap.find(incDirs);
            if (it != m_SystemHeadersMap.end())
            {
                const StringSet& headers = it->second;
                for (StringSet::iterator it = headers.begin(); it != headers.end(); ++it)
                    files.insert(*it);
            }
        }
    }

    // #include "|
    if (!useSystemHeaders && project)
    {
        const wxArrayString localIncludeDirs = GetLocalIncludeDirs(project, buildTargets);
        for (int i = 0; i < project->GetFilesCount(); ++i)
        {
            ProjectFile* pf = project->GetFile(i);
            if (pf && FileTypeOf(pf->relativeFilename) == ftHeader)
            {
                wxString file = pf->file.GetFullPath();
                if (file.Find(filename) != wxNOT_FOUND)
                {
                    wxString header;
                    for (size_t j = 0; j < localIncludeDirs.GetCount(); ++j)
                    {
                        const wxString& dir = localIncludeDirs[j];
                        if (file.StartsWith(dir))
                        {
                            header = file.Mid(dir.Len());
                            break;
                        }
                    }

                    if (header.IsEmpty())
                    {
                        if (pf->buildTargets != buildTargets)
                            continue;

                        wxFileName fn(file);
                        fn.MakeRelativeTo(curPath);
                        header = fn.GetFullPath();
                    }

                    header.Replace(_T("\\"), _T("/"), true);
                    files.insert(header);
                }
            }
        }
    }

    // popup the auto completion window
    if (!files.empty())
    {
        ed->GetControl()->ClearRegisteredImages();
        ed->GetControl()->AutoCompSetIgnoreCase(false);
        ed->GetControl()->AutoCompSetCancelAtStart(true);
        ed->GetControl()->AutoCompSetFillUps(m_CCFillupChars);
        ed->GetControl()->AutoCompSetChooseSingle(false);
        ed->GetControl()->AutoCompSetAutoHide(true);
        ed->GetControl()->AutoCompSetDropRestOfWord(m_IsAutoPopup ? false : true);
        wxString final;
        GetStringFromSet(final, files, _T(" "));
        final.RemoveLast(); // remove last space
        Manager::Get()->GetLogManager()->DebugLog(F(_T("Get include file count is %d, use time is %d"),
                                                    files.size(), sw.Time()));
        ed->GetControl()->AutoCompShow(pos - lineStartPos - keyPos, final);
    }
}
Almost all of the time is spent in the AutoCompShow call.
This is not a CC problem, but wxScintilla.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: polygon7 on October 12, 2010, 04:17:13 pm
#include " --> worked very fast, but several includes missing
#include <  --> still rather slow [and at the start I always have a lot of entries like this : /32/32/32/32/32/32/32....]
All code in here:
(...)
Almost all of the time is spent in the AutoCompShow call.
This is not a CC problem, but wxScintilla.

Maybe that includes popup should be visible only when user write similar number of letters like
in CC (Editor->CC->Automatically launch when typed # letters) after " or < character and
popup should contain only that items which have the same letters on start as written by user?

Eg.
Editor->CC->Automatically launch when typed # letters is set to 4
and with:
Code
#include <cst|
no popup visible

Code
#include <cstd
popup visible and should contain: "cstdio" (or if there is another file with cstd prefix then it should be in popup too).
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 12, 2010, 04:36:13 pm
This problem does not occur in the Windows platform. :(
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on October 12, 2010, 04:56:06 pm
one thing is for sure :
Code
#include "

doesn't show any options any more from file present in the project's include paths (no idea with target's include paths)
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: oBFusCATed on October 12, 2010, 05:05:01 pm
This problem does not occur in the Windows platform. :(
Probably on windows you have far less header files. On linux many softwares install header files in /usr/include
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 12, 2010, 05:57:00 pm
I just mesured it with wxStopWatch.

There are 22996, that are splitted in tokens by wxStringTokenizer in PlatWX.cpp ListBoxImpl::SetList (caled by ScintillaBase::AutoCompleteStart ), that needs the great amount of time.

Maybe we have to create our own listbox to show the includes ?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: oBFusCATed on October 12, 2010, 07:03:59 pm
Or limit the number to 500 and to put "To many includes found..." at the top of the list :)
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 12, 2010, 08:22:48 pm
More tests:
The problem is not the wxStringTokenizer, but appending the string to the listcontrol.
So limiting the maximal amount, or better kicking in the autocomp-listbox after 2 or more characters (as default, configurable) might be the best solution.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 13, 2010, 03:07:56 am
More tests:
The problem is not the wxStringTokenizer, but appending the string to the listcontrol.
So limiting the maximal amount, or better kicking in the autocomp-listbox after 2 or more characters (as default, configurable) might be the best solution.
Thanks a lot! :D
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 13, 2010, 05:36:29 am
#include " --> worked very fast, but several includes missing
#include <  --> still rather slow [and at the start I always have a lot of entries like this : /32/32/32/32/32/32/32....]
Fixed in r6708. :D
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on October 13, 2010, 08:39:02 am
I can confirm that
Code
#include <
now works fast, and one first has to type a few first characters.

But the
Code
#include "
fails for me.

My test project :
simple console project -> the .cbp and main.cpp are in the same directory. The project has some search/include directories added.
But not even one suggestion pops up, even if I start typing parts of the filename of the header to include.

For further testing purposes, I added 2 more files to the same directory where main.cpp is (and the .cbp) : foo.h and bar.h
Also these 2 don't show up in the suggestion list.

Next I added foo.h to the project, from that moment on the include list "foo.h" appears, but not bar.h [I even added the directory where main.cpp and foo/bar.h live as an include directory explicitly].

This is something that used to work, so I guess somewhere there's still a little glitch.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 13, 2010, 09:04:31 am
Next I added foo.h to the project, from that moment on the include list "foo.h" appears, but not bar.h [I even added the directory where main.cpp and foo/bar.h live as an include directory explicitly].
#include "f // appear f****.h
#include "b // appear b****.h
That's right!
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on October 13, 2010, 09:45:15 am
do you say you also can reproduce this problem or that it works for you ?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 13, 2010, 10:11:23 am
do you say you also can reproduce this problem or that it works for you ?
I can not reproduce you problem.
But i found some new issue.
Please trying r6709.

If there have some issue, please show me by picture?
Because in here, it's works well.

When type #include "c|
Will only appear all headers start with 'c'.

Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 13, 2010, 10:12:47 am
If you type #include "cb|
Will list all the headers that start of "cb", like the picture show.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 13, 2010, 10:16:16 am
And if you want auto completion a local header file, the header file must belong the project.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: killerbot on October 13, 2010, 10:45:35 am
that's better ;-)

I now see again the list of headers from the include paths I added.
And also the list of system headers now also reappear in the list when doing the #include "

One thing up for debate. You say local files don't show up in the list unless they are part of the project. Agreed, but when that directory is however added as an include directory, they should appear.
So in my case I added the directory where main.cpp resides as include directory, but the local, not part of the project, bar.h : doesn't show up. I would think it should ?
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 13, 2010, 11:17:18 am
One thing up for debate. You say local files don't show up in the list unless they are part of the project. Agreed, but when that directory is however added as an include directory, they should appear.
So in my case I added the directory where main.cpp resides as include directory, but the local, not part of the project, bar.h : doesn't show up. I would think it should ?

Quote
wxArrayString& CodeCompletion::GetSystemIncludeDirs(Parser* parser, bool force)
{
    static Parser* lastParser = NULL;
    static wxArrayString incDirs;

    if ((!parser || !force) && parser == lastParser)
        return incDirs;
    else
    {
        incDirs.Clear();
        lastParser = parser;
    }

    cbProject* project = m_NativeParser.GetProjectByParser(parser);
    wxString prjPath;
    if (project)
        prjPath = project->GetCommonTopLevelPath();


    incDirs = parser->GetIncludeDirs();
    for (size_t i = 0; i < incDirs.GetCount();)
    {
        if (incDirs.Last() != wxFILE_SEP_PATH)
            incDirs.Append(wxFILE_SEP_PATH);
        if (project && incDirs.StartsWith(prjPath))
            incDirs.RemoveAt(i);

        else
            ++i;
    }

    return incDirs;
}
It seems can not have both! :(

If you comment two lines of code, we can solve your question.
Code
        if (project && incDirs[i].StartsWith(prjPath))
            incDirs.RemoveAt(i);
However, this will produce more side effects.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Borr on October 14, 2010, 04:23:47 pm
CC not show std::towlower
note: CC show std::tolower
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: ollydbg on October 14, 2010, 04:35:50 pm
CC not show std::towlower
note: CC show std::tolower
hi,can you show the minimal sample code? also the steps to produce the bug.

thx.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: MortenMacFly on October 14, 2010, 09:11:20 pm
CC not show std::towlower
note: CC show std::tolower
Are you kidding??? Does it show or not show std::tolower ???
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: stahta01 on October 14, 2010, 09:22:03 pm
CC not show std::towlower
note: CC show std::tolower
Are you kidding??? Does it show or not show std::tolower ???

Notice the w in one of them, I think wide char version.
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Borr on October 14, 2010, 09:26:57 pm
CC not show std::towlower
note: CC show std::tolower
Are you kidding??? Does it show or not show std::tolower ???

Notice the w in one of them, I think wide char version.

Yes that's right to(W)lower (wide char) missing in the CodeCompletion list
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: MortenMacFly on October 14, 2010, 10:21:23 pm
Notice the w in one of them, I think wide char version.
Yes that's right to(W)lower (wide char) missing in the CodeCompletion list
Right... my fault - I missed that. :oops:
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Jenna on October 14, 2010, 10:29:46 pm
I have towlower.
It's not in std-namespace, but in global namespace.

The declaration is between __BEGIN_NAMESPACE_C99 and __END_NAMESPACE_C99, in cdefs.h is the following comment:
Code
/* The standard library needs the functions from the ISO C90 standard
   in the std namespace.  At the same time we want to be safe for
   future changes and we include the ISO C99 code in the non-standard
   namespace __c99.  The C++ wrapper header take case of adding the
   definitions to the global namespace.  */
So I think it is correctly found in global namespace, and it can (and should ? ) be used without std:: ( at least here it works fine).
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Loaden on October 15, 2010, 04:43:04 am
file: lib\gcc\i686-mingw32\4.4.4\include\c++\cwctype
Quote
#pragma GCC system_header

#include <bits/c++config.h>

#if _GLIBCXX_HAVE_WCTYPE_H
#include <wctype.h>
#endif

#ifndef _GLIBCXX_CWCTYPE
#define _GLIBCXX_CWCTYPE 1

// Get rid of those macros defined in <wctype.h> in lieu of real functions.
#undef iswalnum
#undef iswalpha
#if _GLIBCXX_HAVE_ISWBLANK
# undef iswblank
#endif
#undef iswcntrl
#undef iswctype
#undef iswdigit
#undef iswgraph
#undef iswlower
#undef iswprint
#undef iswpunct
#undef iswspace
#undef iswupper
#undef iswxdigit
#undef towctrans
#undef towlower
#undef towupper
#undef wctrans
#undef wctype

#if _GLIBCXX_USE_WCHAR_T

_GLIBCXX_BEGIN_NAMESPACE(std)

  using ::wctrans_t;
  using ::wctype_t;
  using ::wint_t;

  using ::iswalnum;
  using ::iswalpha;
#if _GLIBCXX_HAVE_ISWBLANK
  using ::iswblank;
#endif
  using ::iswcntrl;
  using ::iswctype;
  using ::iswdigit;
  using ::iswgraph;
  using ::iswlower;
  using ::iswprint;
  using ::iswpunct;
  using ::iswspace;
  using ::iswupper;
  using ::iswxdigit;
  using ::towctrans;
  using ::towlower;
  using ::towupper;
  using ::wctrans;
  using ::wctype;

_GLIBCXX_END_NAMESPACE


#endif //_GLIBCXX_USE_WCHAR_T

test demo:
Code
void test() {}

namespace qp
{
    using ::test;
}

int main()
{
    qp::test();
    return 0;
}
Title: Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
Post by: Borr on October 15, 2010, 07:04:38 am
I have towlower.
It's not in std-namespace, but in global namespace.
So I think it is correctly found in global namespace, and it can (and should ? ) be used without std:: ( at least here it works fine).

sorry, it's my mistake. thanks for the reply