The 25 september 2010 build (6634) CODECOMPLETION BRANCH version is out.

[Loaden]

@ Loaden:
I test the patch and give you feedback.

First endless-loop seems to be fixed, now we get one some lines later.

Code

ReplaceBufferForReparse() : <FROM>printf<TO>printk
ReplaceBufferForReparse() : <FROM>printk<TO>printf

tk->m_Type is printf and token is printk and vice versa.

Can this be handled in ReplaceBufferForReparse, so it only has to be done in one place ?

This seems very strange! Jens, could you give me more information?

[Jenna]

There are several headers with #define printf printk and some others with #define printk printf.

As before I am not able to strip the sources down to a small project (the actual one uses 47 MB if it is 7zipped).

After commen ting out all #define printf's I get:

Code

ReplaceBufferForReparse() : <FROM>printk<TO>printf
ReplaceBufferForReparse() : <FROM>(             printf<TO>(args...)
ReplaceBufferForReparse() : <FROM>args...)(args)<TO>            printk(args)
ReplaceBufferForReparse() : <FROM>printk<TO>printf
ReplaceBufferForReparse() : <FROM>(             printf<TO>(args...)
ReplaceBufferForReparse() : <FROM>args...)(args)<TO>            printk(args)
ReplaceBufferForReparse() : <FROM>printk<TO>printf
ReplaceBufferForReparse() : <FROM>(             printf<TO>(args...)

endless as before, m_TokenIndex does not change if I set the brakpoint before ReplaceBufferForReparse(tk->m_Type, false); .

[Loaden]

@ Loaden:
I test the patch and give you feedback.

First endless-loop seems to be fixed, now we get one some lines later.

Code

ReplaceBufferForReparse() : <FROM>printf<TO>printk
ReplaceBufferForReparse() : <FROM>printk<TO>printf

tk->m_Type is printf and token is printk and vice versa.

Can this be handled in ReplaceBufferForReparse, so it only has to be done in one place ?

Thanks! This code lead CB endless loop.
I will fix it soon.

Code

#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif

[ollydbg]

Thanks! This code lead CB endless loop.
I will fix it soon.

Code

#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif

But the code above does not compile.

[Loaden]

Thanks! This code lead CB endless loop.
I will fix it soon.

Code

#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif

But the code above does not compile.

But we still have to avoid all possible infinite loop. :lol:

[Jenna]

Thanks! This code lead CB endless loop.
I will fix it soon.

Code

#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif

But the code above does not compile.

The kernel code surely does not compile also (all files added to a C::B project without running any configuration-scripts and without using makefiles), but it leads to an endless loop when parsing the sources and that should (of course) not happen, even if the code has errors.

[ollydbg]

The kernel code surely does not compile also (all files added to a C::B project without running any configuration-scripts and without using makefiles), but it leads to an endless loop when parsing the sources and that should (of course) not happen, even if the code has errors.

But we still have to avoid all possible infinite loop. :lol:

Yes, 100% agree.
Maybe, we could limit the macro expansion level to avoid the recursive infinite loop.
Eg, once a macro definition is used to do macro expansion, it will never be used again in deeper level expansion.

[Loaden]

Maybe, we could limit the macro expansion level to avoid the recursive infinite loop.

Agreed! This is a good idea.

[Loaden]

The kernel code surely does not compile also (all files added to a C::B project without running any configuration-scripts and without using makefiles), but it leads to an endless loop when parsing the sources and that should (of course) not happen, even if the code has errors.

Hi, Jens, Please trying this patch?

[Jenna]

The kernel code surely does not compile also (all files added to a C::B project without running any configuration-scripts and without using makefiles), but it leads to an endless loop when parsing the sources and that should (of course) not happen, even if the code has errors.

Hi, Jens, Please trying this patch?

The endless loops do no longer occur.

Parsing takes a little more than 28 minutes, that's very long, but I get more than 1 million tokens.
One problem is the amount of memory used (more than 2 GB).
And it seems parsing is not correct, I get some strange tokens in symbol browser (see the attached picture).
I am currently uploading a 7z-file, that contains the tokens tree and the outher lists from cc debug tool.
That will take some time (just ISDN here), because it's about 18 MB (105 MB tokens tree).

Another issue (not related to this), is that the symboslbrowser stays empty, if the project is opened without any shown editor.
After opening any file the symbols-browser gets filled.

[Jenna]

cc debug-info: http://apt.jenslody.de/cc/cc_debug.7z

As written before, the test-project is generated from the (debian) linux-kernel sources, revision 2.6.29.
No build-options or defines, just all files from sources added to an empty project.

[Loaden]

cc debug-info: http://apt.jenslody.de/cc/cc_debug.7z

As written before, the test-project is generated from the (debian) linux-kernel sources, revision 2.6.29.
No build-options or defines, just all files from sources added to an empty project.

Thank Jens! I will look into it carefully.

[Loaden]

Another issue (not related to this), is that the symboslbrowser stays empty, if the project is opened without any shown editor.
After opening any file the symbols-browser gets filled.

Confirmed!

EDIT: It's can confirmed only Linux, and in Windows, no have this issue.
Here:

bool Parser::Done()
{
    wxCriticalSectionLocker locker(s_ParserCritical);
    bool done =    m_UpFrontHeaders.IsEmpty()
                && m_SystemUpFrontHeaders.IsEmpty()
                && m_BatchParseFiles.IsEmpty()
                && m_PredefinedMacros.IsEmpty()
                && !m_NeedMarkFileAsLocal // When parsing end, this value be true always in Linux/GCC4.5.1!
                && m_PoolTask.empty()
                && m_Pool.Done();
    return done;
}

[Loaden]

answer to these questions:

Parsing takes a little more than 28 minutes, that's very long, but I get more than 1 million tokens.

1, the batch parsing time is too long:
the Linux source code you supplied contains a lot of code snippet which has grammar errors like

Code

#define AAA(x) BBB(x)
#define BBB(y) AAA(y)

#if AAA(1) && BBB(2)
void fly();
#else
void good();
#endif

These code will increase the loop time(at least 100 times add compare to normal code), so the performace is bad.

And it seems parsing is not correct, I get some strange tokens in symbol browser (see the attached picture).

2. parsing error, wrong tokens:
This was still due to the highly mixed pre-processor code, our CC only do a format match, like

"AAA BBB();" ---> BBB is a function.
"AAA BBB;"  ---> BBB is a variable.

Thus, CC is not so smart to detect and handle grammar errors. (the only method is do a "full pre-processor" before parsing)

[Jenna]

This sounds reasonable.
Nevertheless, there are tokens that contain (or even begin with) whitespace, that means these tokens are invaild, so they could be (should be ?) securely removed.