New code completion remarks/issues

[ollydbg]

Ok, tiny fixed with ReadClsNames in parserthread.
for broken codes below:

#include <iostream>

using namespace std;
typedef class qq
{
    int x;
    int y;

}xx
int main()
{
    cout << "Hello world!" << endl;
    return 0;
}

Note:This code can't be compiled in G++.

[blueshake]

Exactly,but I refer to visual assist,its parser do what the patch do.

[ollydbg]

Exactly,but I refer to visual assist,its parser do what the patch do.

I have applied your patch in my local copy, it works. . Thanks for your effort!

[blueshake]

It seems

Code

Token* tdef = DoAddToken(tkTypedef, components.front(), lineNr, 0, 0, args);

can not work for the test codes below.

Code

#include <iostream>
#include <iostream>
using namespace std;
std::vector<string> ssss;
#define  xxxxxxxxx 12e
typedef class qq
{
    int x;
    int y;
}eeee;
typedef unsigned int wwwwwwwww;
www|
int main()
{

    cout << "Hello world!" << endl;
    return 0;
}

In the "|" position , the suggestion list flash and can not show.If I comment the statement

Code

typedef unsigned int wwwwwwwww;

,it works again.

[ollydbg]

@blueshake
That's something related in the nativeparser.cpp, and the code you cited. I will exam that.

[blueshake]

@blueshake
That's something related in the nativeparser.cpp, and the code you cited. I will exam that.

Nice to hear that.Thanks.

[ollydbg]

I can only understand a little about the FindAIMatches() function.

For example, your test code:

Code

typedef unsigned int wwwww;

wwww|

You will see in the symbol tree, the "wwwww" is a global "typedef". which is quite different if your code is like below:

Code

typedef class qq
{
    int x;
    int y;
}eeee;

then, "eeee" in the symbol tree is of type "class".


The piece of code in FindAIMatches()

Code

if (local_result.size() == 1)
    {
        int id = *local_result.begin();
        Token* token = tree->at(id);

        if (token->m_TokenKind == tkTypedef)
        {
            std::queue<ParserComponent> type_components;
            BreakUpComponents(parser, token->m_ActualType, type_components);

            while(!components.empty())
            {
                ParserComponent comp = components.front();
                components.pop();
                type_components.push(comp);
            }

    #if DEBUG_CC_AI
            if (s_DebugSmartSense)
            #if wxCHECK_VERSION(2, 9, 0)
                Manager::Get()->GetLogManager()->DebugLog(F(_T("Replacing %s to %s"), token->m_Name.wx_str(), token->m_ActualType.wx_str()));
            #else
                Manager::Get()->GetLogManager()->DebugLog(F(_T("Replacing %s to %s"), token->m_Name.c_str(), token->m_ActualType.c_str()));
            #endif
    #endif
            return FindAIMatches(parser, type_components, result, parentTokenIdx, noPartialMatch, caseSensitive, use_inheritance, kindMask, search_scope);
        }

    }

this means, when you only get one match in the current context, then for example, you are entering

Code

www

This time, the only matched string is "wwwww" will be replace with "unsigned int" and the total line will be reparsed. (which means the FindAIMatches will be called iteratively).

 

But I think at this test code, this typedef replacement is useless. I can't find a test case that this replacement is meaningful.

Can someone explain more about this? Thanks.

[MortenMacFly]

Guys, I am a bit lost and trying to catch up with all the posts. Am I rioght, that these two patches have to be applied together?

Patch 1:

Code

Index: tokenizer.cpp
===================================================================
--- tokenizer.cpp	(revision 5834)
+++ tokenizer.cpp	(working copy)
@@ -216,17 +216,22 @@
 
 bool Tokenizer::SkipToCharBreak()
 {
-  if (PreviousChar() != '\\')
-      return true;
-  else
-  {
-      // check for "\\"
-      if (   ((m_TokenIndex - 2) >= 0)
-          && ((m_TokenIndex - 2) <= m_BufferLen)
-          && (m_Buffer.GetChar(m_TokenIndex - 2) == '\\') )
-          return true;
-  }
-  return false;
+    if (PreviousChar() != '\\')
+        return true;
+    else
+    {
+        // check for "\\"
+        unsigned int numBackslash = 2;
+        while(   ((m_TokenIndex - numBackslash) >= 0)
+            && ((m_TokenIndex - numBackslash) <= m_BufferLen)
+            && (m_Buffer.GetChar(m_TokenIndex - numBackslash) == '\\') )
+            ++numBackslash;
+        if(numBackslash%2==1) // number of back slash(include current char ) is even
+            return true;     // eg: "\""
+        else
+            return false;    // eg: "\\""
+    }
+    return false;
 }

Patch 2:

Code

Index: src/plugins/codecompletion/parser/parserthread.cpp
===================================================================
--- src/plugins/codecompletion/parser/parserthread.cpp	(revision 5825)
+++ src/plugins/codecompletion/parser/parserthread.cpp	(working copy)
@@ -1768,7 +1768,7 @@
     Manager::Get()->GetLogManager()->DebugLog(F(_("HandleTypedef() : Adding typedef: name='%s', ancestor='%s'"), components.front().c_str(), ancestor.c_str()));
 #endif
 //    Token* tdef = DoAddToken(tkTypedef, components.front(), lineNr, 0, 0, args);
-    Token* tdef = DoAddToken(tkClass, components.front(), lineNr, 0, 0, args);
+    Token* tdef = DoAddToken(tkTypedef, components.front(), lineNr, 0, 0, args);
     if (tdef)
     {
         if (!is_function_pointer)
@@ -1796,7 +1796,7 @@
                 m_Tokenizer.UngetToken();
                 break;
             }
-        else if (wxIsalpha(current.GetChar(0)))
+        else if (wxIsalpha(current.GetChar(0)) && (m_Tokenizer.PeekToken() == ParserConsts::semicolon || m_Tokenizer.PeekToken() == ParserConsts::comma))
         {
 #if PARSERTHREAD_DEBUG_OUTPUT
             Manager::Get()->GetLogManager()->DebugLog(F(_T("ReadClsNames() : Adding variable '%s' as '%s' to '%s'"), current.c_str(), m_Str.c_str(), (m_pLastParent?m_pLastParent->m_Name.c_str():_T("<no-parent>"))));
@@ -1811,7 +1811,10 @@
             }
 
         }
-        else // unexpected
+        else// unexpected
+        {
+            m_Tokenizer.UngetToken();
             break;
+        }
     }
 }

I think the function name "SkipToCharBreak" is confusion, because it doesn't really do a skip.
So, I suggest using another name like :  IsRealCharBreak or IsCharBreak or IsEscapeChar

That's right and probably was a typo. I'll change that certainly.

[ollydbg]

@MortenMacFly
Yes, the two patches can both be applied.

But, blueshake's patch of changing the code :

Code

-    Token* tdef = DoAddToken(tkClass, components.front(), lineNr, 0, 0, args);
+    Token* tdef = DoAddToken(tkTypedef, components.front(), lineNr, 0, 0, args);

should be reconsidered, because a small bug he states in this post:
http://forums.codeblocks.org/index.php/topic,11187.msg76788.html#msg76788

[blueshake]

hello,another bad news.
It seems some bugs hide in the codecompletion.In the file codecompletion.cpp,in this place:

Code

            // Remove duplicate items
            for (size_t i=0; i<items.Count()-1; i++)
                if (items.Item(i)==items.Item(i+1))
                    items.RemoveAt(i);
Manager::ge|

when you type Mana,yes,the suggestion show,then you type ::,yes work again,but when you type ge,the issue come out,the suggestion list flash again,and not show.what is going on here,the suggestion list is try to show,but cancle by something.

Note:
I just use the latest cc,change nothing.
Edit:

Code

Manager::Get()->|

And in the "|" position, the cc not work.

[ollydbg]

hello,another bad news.
It seems some bugs hide in the codecompletion.In the file codecompletion.cpp,in this place:

Code

            // Remove duplicate items
            for (size_t i=0; i<items.Count()-1; i++)
                if (items.Item(i)==items.Item(i+1))
                    items.RemoveAt(i);
Manager::ge|

when you type Mana,yes,the suggestion show,then you type ::,yes work again,but when you type ge,the issue come out,the suggestion list flash again,and not show.what is going on here,the suggestion list is try to show,but cancle by something.

Note:
I just use the latest cc,change nothing.
Edit:

Code

Manager::Get()->|

And in the "|" position, the cc not work.

I can confirm both the two bugs.
Is it possible to create a simple test case. So that I can debug it. Currently, Opening the codeblocks's whole source code is not necessary.

Also: I think it's time to create some standard test code for codecompletion now. Any comments?

Also 1: @morten, in the current CC code, there are many code using the preprocessor guard.like:

Code

#if PARSERTHREAD_DEBUG_OUTPUT
            Manager::Get()->GetLogManager()->DebugLog(F(_T("HandlePreprocessorBlocks() : Special case \"#if 0\" not skipped.")));
#endif

Is there any way we can just clean up these code, and use something like this:

Code

#if PARSERTHREAD_DEBUG_OUTPUT
#define DebugOutput(a,b,c)  Manager::Get()->GetLogManager()->DebugLog(a,b,c)
#else
#define DebugOutput(a,b,c)  
#endif

So, the the source code, we can only use this:

Code

DebugOutput(F(_T("HandlePreprocessorBlocks() : Special case \"#if 0\" not skipped.")));

This will make the source code more clean.


Also 2:

There are many variable names which are not following the source style guard. like:

Code

    Parser parser(this);
    parser.ParseBufferForFunctions(control->GetTextRange(0, pos));

    wxArrayString funcs;
    TokensTree* tmptree = parser.GetTempTokens();

    // look for implementation functions that enclose our current line number
    for(size_t i = 0; i < tmptree->size();i++)

the tmptree should be tmpTree.

Also, there are many other variables should be renamed. such as:

Code

// find current function's namespace so we can include local scope's tokens
    // we ' ll get the function's token (all matches) and add its parent namespace
    TokenIdxSet scope_result;
    TokenIdxSet proc_result;
    if (FindCurrentFunctionToken(editor, proc_result) != 0)
    {
        for (TokenIdxSet::iterator it = proc_result.begin(); it != proc_result.end(); ++it)
        {
            Token* token = parser->GetTokens()->at(*it);
            if (!token)
                continue;
            scope_result.insert(token->m_ParentIndex);
#if DEBUG_CC_AI
            if (s_DebugSmartSense)
            {
                Token* parent = parser->GetTokens()->at(token->m_ParentIndex);
                Manager::Get()->GetLogManager()->DebugLog(_T("Adding search namespace: ") + (parent ? parent->m_Name : _T("Global namespace")));
            }
#endif
        }
    }

These name should be "scopeResult" and so on...

[ollydbg]

@blueshake:

look at the Get function prototype, it is a static function.

Code

class DLLIMPORT Manager
{
    static Manager* Get();
}

but, in the symbol tree debug dialog, you will see that the

type is Manager *
actual type is Manager

That's may be the cause of bug...

[blueshake]

@blueshake:

look at the Get function prototype, it is a static function.

Code

class DLLIMPORT Manager
{
    static Manager* Get();
}

but, in the symbol tree debug dialog, you will see that the

type is Manager *
actual type is Manager

That's may be the cause of bug...

But it worked before.I updated the local copy,and it don't work.
look at these codes.The static are ignored.

Code

        else if (token==ParserConsts::kw_static ||
            token==ParserConsts::kw_virtual ||
            token==ParserConsts::kw_inline)
        {
            // do nothing (skip it)
        }

[blueshake]

anther issue.

Code

std::string ss;
ss. not work

Code

string ss;
ss.work

parser error:

Code

string ss("abc");
ss.

it should be a variable,but ss is parsed as function.

[attachment deleted by admin]

[MortenMacFly]

Just a tiny remark: For testing if the current modification (a.k.a patch) does not break something else please make sure that after applying the patch do the following:
1.) Open Code::Blocks having the debug log enabled.
2.) Open CodeBlocks.cbp (project files of core and core plugins)
3.) Wait, until the parser has completed.
3.) Check the debug log for the number of tokens, it should look like the following:
Parsing stage done (1570 total parsed files, 92637 tokens in 1 minute(s), 8.922 seconds).
The tokens should be round about 90000 - 100000 (depending on the patch level of C::B).

Furthermore: I am still happy to collect sample projects for testing CC I already have some...