Author Topic: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.  (Read 115513 times)

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #45 on: October 04, 2010, 08:42:52 am »
Another issue (not related to this), is that the symboslbrowser stays empty, if the project is opened without any shown editor.
After opening any file the symbols-browser gets filled.
Confirmed!

EDIT: It's can confirmed only Linux, and in Windows, no have this issue.
Here:
Quote
bool Parser::Done()
{
    wxCriticalSectionLocker locker(s_ParserCritical);
    bool done =    m_UpFrontHeaders.IsEmpty()
                && m_SystemUpFrontHeaders.IsEmpty()
                && m_BatchParseFiles.IsEmpty()
                && m_PredefinedMacros.IsEmpty()
                && !m_NeedMarkFileAsLocal // When parsing end, this value be true always in Linux/GCC4.5.1!
                && m_PoolTask.empty()
                && m_Pool.Done();
    return done;
}
There seem to be two instances of NativeParser with two different variables for m_Parser, one parser returns true for Done() and the other one returns false.
The one that returns true is called in CodeCompletion::OnParserEnd and the other one is the one that catches the PARSER_END event in NativeParser::OnParserEnd .

Offline killerbot

  • Administrator
  • Lives here!
  • *****
  • Posts: 5489
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #46 on: October 04, 2010, 09:53:47 am »
please have a look at this. It seems a little regression got introduced : http://forums.codeblocks.org/index.php/topic,13413.msg90463.html#msg90463

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #47 on: October 04, 2010, 10:11:26 am »
please have a look at this. It seems a little regression got introduced : http://forums.codeblocks.org/index.php/topic,13413.msg90463.html#msg90463
OK, I will look into it, and hopefully fixed soon.

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #48 on: October 04, 2010, 04:19:28 pm »
There seem to be two instances of NativeParser with two different variables for m_Parser, one parser returns true for Done() and the other one returns false.
The one that returns true is called in CodeCompletion::OnParserEnd and the other one is the one that catches the PARSER_END event in NativeParser::OnParserEnd .
Fixed in r6665, thanks!

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #49 on: October 05, 2010, 08:38:35 am »
Nevertheless, there are tokens that contain (or even begin with) whitespace, that means these tokens are invaild, so they could be (should be ?) securely removed.
I found a reason, in file "/home/jens/kernel-tmp/linux-source-2.6.29/sound/aoa/codecs/tas.c", there has a struct named tas:
Code
struct tas {
    struct aoa_codec    codec;
    struct i2c_client    *i2c;
    u32            mute_l:1, mute_r:1 ,
                controls_created:1 ,
                drc_enabled:1,
                hw_enabled:1;
    u8            cached_volume_l, cached_volume_r;
    u8            mixer_l[3], mixer_r[3];
    u8            bass, treble;
    u8            acr;
    int            drc_range;
    /* protects hardware access against concurrency from
     * userspace when hitting controls and during
     * codec init/suspend/resume */
    struct mutex        mtx;
};
and in "/home/jens/kernel-tmp/linux-source-2.6.29/arch/cris/include/arch-v32/arch/system.h"
Here a macro define, like this:
Quote
preprocessor #define tas(ptr) (xchg((ptr),1))    [186,42]
So, we will get a error parse result:
Quote
/home/jens/kernel-tmp/linux-source-2.6.29/sound/aoa/codecs/tas.c
preprocessor #define PFX DEVNAME ": "    [1,77]
class class (xchg((ptr),1)) {...}    [83,83]
variable u32 (xchg((ptr),1))::mute_l    [86,0]
variable u32 (xchg((ptr),1))::mute_r    [86,0]
variable u32 (xchg((ptr),1))::controls_created    [87,0]
variable u32 (xchg((ptr),1))::drc_enabled    [88,0]
variable u32 (xchg((ptr),1))::hw_enabled    [89,0]
variable u32u8 (xchg((ptr),1))::cached_volume_l    [90,0]
variable u32u8 (xchg((ptr),1))::cached_volume_r    [90,0]
variable u8 (xchg((ptr),1))::mixer_l    [91,0]
variable u8 (xchg((ptr),1))::mixer_r    [91,0]
variable u8 (xchg((ptr),1))::bass    [92,0]
variable u8 (xchg((ptr),1))::treble    [92,0]
variable u8 (xchg((ptr),1))::acr    [93,0]

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #50 on: October 05, 2010, 08:41:56 am »
Here is a test code snippet:
Code
#define tas(ptr) (xchg((ptr),1))

struct tas {
    struct aoa_codec    codec;
    struct i2c_client    *i2c;
    u32            mute_l:1, mute_r:1 ,
                controls_created:1 ,
                drc_enabled:1,
                hw_enabled:1;
    u8            cached_volume_l, cached_volume_r;
    u8            mixer_l[3], mixer_r[3];
    u8            bass, treble;
    u8            acr;
    int            drc_range;
    /* protects hardware access against concurrency from
     * userspace when hitting controls and during
     * codec init/suspend/resume */
    struct mutex        mtx;
};

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #51 on: October 05, 2010, 09:19:06 am »
Nevertheless, there are tokens that contain (or even begin with) whitespace, that means these tokens are invaild, so they could be (should be ?) securely removed.
This issue shoud be fixed in r6670. :D

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #52 on: October 05, 2010, 07:02:26 pm »
Next endless loop, this time with linux kernel-sources 2.6.35.
It happens in parserthread.cpp:2226 ( m_Tokenizer.ReplaceBufferForReparse(m_Tokenizer.GetActualContextForMacro(tk)); ).

Snippet of debug-output parserthread and tokenizer:

Code
DoParse() : Loop:m_Str='', token='dma_addr_t'
ReadParentheses(): (* map_single), line=355
HandleMacro() : Adding token 'dma_addr_t' (peek='(* map_single)')
DoAddToken() : Found token (parent).
GetActualTokenType() : Searching within m_Str=''
GetActualTokenType() : Compensated m_Str=''
GetActualTokenType() : Returning ''
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'dma_addr_t' (490410), type '', actual ''. Parent is esp_driver_ops (490406)
The replaced actual context are 'dma_addr_t'.
ReplaceBufferForReparse() : <FROM>dma_addr_t<TO>dma_addr_t
DoParse() : Loop:m_Str='', token='dma_addr_t'
ReadParentheses(): (* map_single), line=355
HandleMacro() : Adding token 'dma_addr_t' (peek='(* map_single)')
DoAddToken() : Found token (parent).
GetActualTokenType() : Searching within m_Str=''
GetActualTokenType() : Compensated m_Str=''
GetActualTokenType() : Returning ''
DoAddToken() : Prepending ''
DoAddToken() : Added/updated token 'dma_addr_t' (490410), type '', actual ''. Parent is esp_driver_ops (490406)
The replaced actual context are 'dma_addr_t'.
ReplaceBufferForReparse() : <FROM>dma_addr_t<TO>dma_addr_t

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #53 on: October 06, 2010, 06:17:02 am »
Next endless loop, this time with linux kernel-sources 2.6.35.
Fixed in r6671.
Thanks!

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #54 on: October 06, 2010, 09:41:07 am »
Next endless loop, this time with linux kernel-sources 2.6.35.
Fixed in r6671.
Thanks!

Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Thank you !

And also the annoying issue, that the symbols-browser is initially empty if no file is opened, seems to be fixed !
« Last Edit: October 06, 2010, 09:46:01 am by jens »

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5906
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #55 on: October 06, 2010, 10:10:36 am »
Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Hi, jens, Did you remember how long will the batch parse take to parse the whole Linux source, when you did this last time(If I remember correct, maybe one year ago)?



If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #56 on: October 06, 2010, 10:14:50 am »
And also the annoying issue, that the symbols-browser is initially empty if no file is opened, seems to be fixed !
Yes, This issue fixed in r6665.
Quote
* cc_branch: fix parser init error, lead class browser can not correctly refreshed

-------------------------------
M : /branches/codecompletion_refactoring/src/plugins/codecompletion/nativeparser.cpp 

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #57 on: October 06, 2010, 10:19:59 am »
Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).
Hi, Jens, About the batch parsing time, could you shut off the debug log trace, and trying again?
Here:
Quote
#define CC_TOKENIZER_DEBUG_OUTPUT 0
#define CC_PARSERTHREAD_DEBUG_OUTPUT 0
...
If use debug log, will lead batch parsing as long time.

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #58 on: October 06, 2010, 10:21:53 am »
Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).

Hi, jens, Did you remember how long will the batch parse take to parse the whole Linux source, when you did this last time(If I remember correct, maybe one year ago)?

No, and it would not give any answers about incfreasing or decreasing parse time, because this time I parsed 2.6.35 kernel sources, the last time it was 2.6.29 with only 21k files and a little more than a million tokens, but I will test with current trunk and give you feedback.

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.
« Reply #59 on: October 06, 2010, 10:25:08 am »
Works now:
Code
Project 'test' parsing stage done (30453 total parsed files, 1347032 tokens in 42 minute(s), 24.656 seconds).
Hi, Jens, About the batch parsing time, could you shut off the debug log trace, and trying again?
Here:
Quote
#define CC_TOKENIZER_DEBUG_OUTPUT 0
#define CC_PARSERTHREAD_DEBUG_OUTPUT 0
...
If use debug log, will lead batch parsing as long time.

I can do, but it will not make such a big difference I guess, because I have an additional static variable inside the ifdef to switch on debug output while running debugger (by changing the value from false to true).
But I will test anyway.
When running batch output all the time it would last much longer.