The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.

[Jenna]

Another issue (not related to this), is that the symboslbrowser stays empty, if the project is opened without any shown editor.
After opening any file the symbols-browser gets filled.

Confirmed!

EDIT: It's can confirmed only Linux, and in Windows, no have this issue.
Here:

bool Parser::Done()
{
    wxCriticalSectionLocker locker(s_ParserCritical);
    bool done =    m_UpFrontHeaders.IsEmpty()
                && m_SystemUpFrontHeaders.IsEmpty()
                && m_BatchParseFiles.IsEmpty()
                && m_PredefinedMacros.IsEmpty()
                && !m_NeedMarkFileAsLocal // When parsing end, this value be true always in Linux/GCC4.5.1!
                && m_PoolTask.empty()
                && m_Pool.Done();
    return done;
}

There seem to be two instances of NativeParser with two different variables for m_Parser, one parser returns true for Done() and the other one returns false.
The one that returns true is called in CodeCompletion::OnParserEnd and the other one is the one that catches the PARSER_END event in NativeParser::OnParserEnd .

[killerbot]

please have a look at this. It seems a little regression got introduced : http://forums.codeblocks.org/index.php/topic,13413.msg90463.html#msg90463

[Loaden]

please have a look at this. It seems a little regression got introduced : http://forums.codeblocks.org/index.php/topic,13413.msg90463.html#msg90463

OK, I will look into it, and hopefully fixed soon.

[Loaden]

There seem to be two instances of NativeParser with two different variables for m_Parser, one parser returns true for Done() and the other one returns false.
The one that returns true is called in CodeCompletion::OnParserEnd and the other one is the one that catches the PARSER_END event in NativeParser::OnParserEnd .

Fixed in r6665, thanks!

[Loaden]

Nevertheless, there are tokens that contain (or even begin with) whitespace, that means these tokens are invaild, so they could be (should be ?) securely removed.

I found a reason, in file "/home/jens/kernel-tmp/linux-source-2.6.29/sound/aoa/codecs/tas.c", there has a struct named tas:

Code

struct tas {
    struct aoa_codec    codec;
    struct i2c_client    *i2c;
    u32            mute_l:1, mute_r:1 ,
                controls_created:1 ,
                drc_enabled:1,
                hw_enabled:1;
    u8            cached_volume_l, cached_volume_r;
    u8            mixer_l[3], mixer_r[3];
    u8            bass, treble;
    u8            acr;
    int            drc_range;
    /* protects hardware access against concurrency from
     * userspace when hitting controls and during
     * codec init/suspend/resume */
    struct mutex        mtx;
};

and in "/home/jens/kernel-tmp/linux-source-2.6.29/arch/cris/include/arch-v32/arch/system.h"
Here a macro define, like this:

preprocessor #define tas(ptr) (xchg((ptr),1))    [186,42]

So, we will get a error parse result:

/home/jens/kernel-tmp/linux-source-2.6.29/sound/aoa/codecs/tas.c
preprocessor #define PFX DEVNAME ": "    [1,77]
class class (xchg((ptr),1)) {...}    [83,83]
variable u32 (xchg((ptr),1))::mute_l    [86,0]
variable u32 (xchg((ptr),1))::mute_r    [86,0]
variable u32 (xchg((ptr),1))::controls_created    [87,0]
variable u32 (xchg((ptr),1))::drc_enabled    [88,0]
variable u32 (xchg((ptr),1))::hw_enabled    [89,0]
variable u32u8 (xchg((ptr),1))::cached_volume_l    [90,0]
variable u32u8 (xchg((ptr),1))::cached_volume_r    [90,0]
variable u8 (xchg((ptr),1))::mixer_l    [91,0]
variable u8 (xchg((ptr),1))::mixer_r    [91,0]
variable u8 (xchg((ptr),1))::bass    [92,0]
variable u8 (xchg((ptr),1))::treble    [92,0]
variable u8 (xchg((ptr),1))::acr    [93,0]

[Loaden]

Here is a test code snippet:

Code

#define tas(ptr) (xchg((ptr),1))

struct tas {
    struct aoa_codec    codec;
    struct i2c_client    *i2c;
    u32            mute_l:1, mute_r:1 ,
                controls_created:1 ,
                drc_enabled:1,
                hw_enabled:1;
    u8            cached_volume_l, cached_volume_r;
    u8            mixer_l[3], mixer_r[3];
    u8            bass, treble;
    u8            acr;
    int            drc_range;
    /* protects hardware access against concurrency from
     * userspace when hitting controls and during
     * codec init/suspend/resume */
    struct mutex        mtx;
};

[Loaden]

Nevertheless, there are tokens that contain (or even begin with) whitespace, that means these tokens are invaild, so they could be (should be ?) securely removed.

This issue shoud be fixed in r6670.

[Jenna]

Next endless loop, this time with linux kernel-sources 2.6.35.
It happens in parserthread.cpp:2226 ( m_Tokenizer.ReplaceBufferForReparse(m_Tokenizer.GetActualContextForMacro(tk)); ).

Snippet of debug-output parserthread and tokenizer:

Code

DoParse() : Loop:m_Str='', token='dma_addr_t'
ReadParentheses(): (* map_single), line=355
HandleMacro() : Adding token 'dma_addr_t' (peek='(* map_single)')
DoAddToken() : Found token (parent).
GetActualTokenType() : Searching within m_Str=''
GetActualTokenType() : Compensated m_Str=''
GetActualTokenType() : Returning ''
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'dma_addr_t' (490410), type '', actual ''. Parent is esp_driver_ops (490406)
The replaced actual context are 'dma_addr_t'.
ReplaceBufferForReparse() : <FROM>dma_addr_t<TO>dma_addr_t
DoParse() : Loop:m_Str='', token='dma_addr_t'
ReadParentheses(): (* map_single), line=355
HandleMacro() : Adding token 'dma_addr_t' (peek='(* map_single)')
DoAddToken() : Found token (parent).
GetActualTokenType() : Searching within m_Str=''
GetActualTokenType() : Compensated m_Str=''
GetActualTokenType() : Returning ''
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'dma_addr_t' (490410), type '', actual ''. Parent is esp_driver_ops (490406)
The replaced actual context are 'dma_addr_t'.
ReplaceBufferForReparse() : <FROM>dma_addr_t<TO>dma_addr_t

[Loaden]

Next endless loop, this time with linux kernel-sources 2.6.35.

Fixed in r6671.
Thanks!

[Jenna]

Next endless loop, this time with linux kernel-sources 2.6.35.

Fixed in r6671.
Thanks!

Works now:

Code

Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Thank you !

And also the annoying issue, that the symbols-browser is initially empty if no file is opened, seems to be fixed !

[ollydbg]

Works now:

Code

Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Hi, jens, Did you remember how long will the batch parse take to parse the whole Linux source, when you did this last time(If I remember correct, maybe one year ago)?

[Loaden]

And also the annoying issue, that the symbols-browser is initially empty if no file is opened, seems to be fixed !

Yes, This issue fixed in r6665.

* cc_branch: fix parser init error, lead class browser can not correctly refreshed

-------------------------------
M : /branches/codecompletion_refactoring/src/plugins/codecompletion/nativeparser.cpp 

[Loaden]

Works now:

Code

Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Hi, Jens, About the batch parsing time, could you shut off the debug log trace, and trying again?
Here:

#define CC_TOKENIZER_DEBUG_OUTPUT 0
#define CC_PARSERTHREAD_DEBUG_OUTPUT 0
...

If use debug log, will lead batch parsing as long time.

[Jenna]

Works now:

Code

Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Hi, jens, Did you remember how long will the batch parse take to parse the whole Linux source, when you did this last time(If I remember correct, maybe one year ago)?

No, and it would not give any answers about incfreasing or decreasing parse time, because this time I parsed 2.6.35 kernel sources, the last time it was 2.6.29 with only 21k files and a little more than a million tokens, but I will test with current trunk and give you feedback.

[Jenna]

Works now:

Code

Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Hi, Jens, About the batch parsing time, could you shut off the debug log trace, and trying again?
Here:

#define CC_TOKENIZER_DEBUG_OUTPUT 0
#define CC_PARSERTHREAD_DEBUG_OUTPUT 0
...

If use debug log, will lead batch parsing as long time.

I can do, but it will not make such a big difference I guess, because I have an additional static variable inside the ifdef to switch on debug output while running debugger (by changing the value from false to true).
But I will test anyway.
When running batch output all the time it would last much longer.