Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => CodeCompletion redesign => Topic started by: oBFusCATed on January 20, 2013, 11:02:31 pm

Title: Patch 3407: correctly parse c++98 enums
Post by: oBFusCATed on January 20, 2013, 11:02:31 pm
Can someone with better knowledge review this patch? Is it possible to add tests for it?
This is a major problem of the current parser and I'd be happy if it is fixed.

I'll apply the patch on my work version of C::B and will give it some testing.

Probably C++11's strong enums (fake) parsing will be broken by this patch, but for me supporting the old enums is preferable.

https://developer.berlios.de/patch/?func=detailpatch&patch_id=3407&group_id=5358
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: killerbot on January 21, 2013, 07:33:00 am
please keep the enum class also working.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: oBFusCATed on January 21, 2013, 09:28:52 am
please keep the enum class also working.
Does they really work or it is just illusion, because of the bug? :)
Apply the patch and test. Then provide feedback...

p.s. I knew you'll would post this :)
p.p.s. c++11 enums are still horrible, the improvement is close to none. (don't flame on this, I don't care)
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: MortenMacFly on January 21, 2013, 04:49:25 pm
please keep the enum class also working.
Dunno what that means?! What class?

Does they really work or it is just illusion, because of the bug? :)
Applied in my local copy... test cases can be added here, if you like:
https://svn.berlios.de/svnroot/repos/codeblocks/trunk/src/plugins/codecompletion/testing/structs_typedefs.cpp (https://svn.berlios.de/svnroot/repos/codeblocks/trunk/src/plugins/codecompletion/testing/structs_typedefs.cpp)
...renaming it to something like structs_typedefs_enums.cpp ... harhar...

Will report back...
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: killerbot on January 21, 2013, 06:03:09 pm
I will test the tomorrow, to see if it still works for enum class.
Will report back ...
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: thomas on January 21, 2013, 07:38:07 pm
please keep the enum class also working.
Dunno what that means?! What class?
C++11 strongly typed enumerations: enum class.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: killerbot on January 21, 2013, 08:18:46 pm
I have tested : this patch is NOT acceptable. It breaks C++11 enum classes.

I think it is better to hand this problem over to our CC gurus, and have them, fix it properly.

Code
#include <iostream>

struct B
{
enum Eb
{
ebAsd1, ebAsd2, ebAsd3,
};

enum{ unnamedEnumerator };

enum class Fruit2
{
Apple,
Strawberry,
Orange,
Cherry
};

};

enum class Fruit
{
Apple,
Strawberry,
Orange,
Cherry
};

enum class Color
{
Red,
Orange,
Green
};

int main()
{

    return 0;
}

Fruit2 is broken, their values end up in the higher scope, which is incorrect. This workaround (patch) should not trigger when enum class.

To which CC guru do we send this ?
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: oBFusCATed on January 21, 2013, 09:26:24 pm
To which CC guru do we send this ?
Ollydbg he is the only one:)

Keep in mind that we don't claim support for C++11, also the normal C++98/C style enums are way more common.
The old code worked by chance I suppose. So if you ask me and there are no other blocker it is better.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: p2rkw on January 21, 2013, 10:16:14 pm
How about this:
Code
Index: src/plugins/codecompletion/parser/parserthread.cpp
===================================================================
--- src/plugins/codecompletion/parser/parserthread.cpp (wersja 8785)
+++ src/plugins/codecompletion/parser/parserthread.cpp (kopia robocza)
@@ -2263,10 +2282,13 @@
     // enums have the following rough definition:
     // enum [xxx] { type1 name1 [= 1][, [type2 name2 [= 2]]] };
     bool isUnnamed = false;
+    bool isEnumClass = false;
     int lineNr = m_Tokenizer.GetLineNumber();
     wxString token = m_Tokenizer.GetToken();
-    if (token == ParserConsts::kw_class)
+    if (token == ParserConsts::kw_class){
         token = m_Tokenizer.GetToken();
+        isEnumClass = true;
+    }
     if (token.IsEmpty())
         return;
 
@@ -2357,7 +2379,8 @@
             {
                 Token* lastParent = m_LastParent;
                 m_LastParent = newEnum;
-                DoAddToken(tkEnumerator, token, m_Tokenizer.GetLineNumber());
+                Token* enumerator = DoAddToken(tkEnumerator, token, m_Tokenizer.GetLineNumber());
+                enumerator->m_Scope = isEnumClass ? tsPrivate : tsPublic;
                 m_LastParent = lastParent;
             }
             if (peek==ParserConsts::colon)
Index: src/plugins/codecompletion/nativeparser_base.h
===================================================================
--- src/plugins/codecompletion/nativeparser_base.h (wersja 8785)
+++ src/plugins/codecompletion/nativeparser_base.h (kopia robocza)
@@ -396,14 +396,16 @@
     // for GenerateResultSet()
     bool AddChildrenOfUnnamed(TokenTree* tree, const Token* parent, TokenIdxSet& result)
     {
-        if (parent->m_TokenKind == tkClass && parent->m_Name.StartsWith(g_UnnamedSymbol))
+        if (((parent->m_TokenKind & (tkClass | tkEnum)) != 0)
+            && parent->m_Name.StartsWith(g_UnnamedSymbol))
         {
             // add all its children
             for (TokenIdxSet::const_iterator it = parent->m_Children.begin();
                                              it != parent->m_Children.end(); ++it)
             {
                 Token* tokenChild = tree->at(*it);
-                if (tokenChild)
+                if (tokenChild &&
+                    (parent->m_TokenKind == tkClass || tokenChild->m_Scope != tsPrivate))
                     result.insert(*it);
             }
 
@@ -411,7 +413,25 @@
         }
         return false;
     }
+   
+    bool AddChildrenOfEnum(TokenTree* tree, const Token* parent, TokenIdxSet& result)
+    {
+        if (parent->m_TokenKind == tkEnum)
+        {
+            // add all its children
+            for (TokenIdxSet::const_iterator it = parent->m_Children.begin();
+                                             it != parent->m_Children.end(); ++it)
+            {
+                Token* tokenChild = tree->at(*it);
+                if (tokenChild && tokenChild->m_Scope != tsPrivate)
+                    result.insert(*it);
+            }
 
+            return true;
+        }
+        return false;
+    }
+
     // for GenerateResultSet()
     bool MatchText(const wxString& text, const wxString& target, bool caseSens, bool isPrefix)
     {
Index: src/plugins/codecompletion/nativeparser_base.cpp
===================================================================
--- src/plugins/codecompletion/nativeparser_base.cpp (wersja 8785)
+++ src/plugins/codecompletion/nativeparser_base.cpp (kopia robocza)
@@ -1318,7 +1318,11 @@
                 if (!token)
                     continue;
                 if ( !AddChildrenOfUnnamed(tree, token, result) )
+                {
                     result.insert(*it);
+                    AddChildrenOfEnum(tree, token, result);
+                }
+                   
             }
 
             tree->RecalcInheritanceChain(parent);
@@ -1334,7 +1338,10 @@
                     if (!token)
                         continue;
                     if ( !AddChildrenOfUnnamed(tree, token, result) )
+                    {
                         result.insert(*it2);
+                        AddChildrenOfEnum(tree, token, result);
+                    }
                 }
             }
         }

Title: Re: Patch 3407: correctly parse c++98 enums
Post by: oBFusCATed on January 21, 2013, 10:24:50 pm
p2rkw: Can you try to add some tests? See Morten's comment.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: killerbot on January 21, 2013, 11:10:42 pm
YES : this updated patch seems to work correctly    ;D
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: MortenMacFly on January 22, 2013, 07:55:39 am
p2rkw: Can you try to add some tests? See Morten's comment.
I've done it. Please test against them:
https://svn.berlios.de/svnroot/repos/codeblocks/trunk/src/plugins/codecompletion/testing/structs_typedefs_enums.cbp (https://svn.berlios.de/svnroot/repos/codeblocks/trunk/src/plugins/codecompletion/testing/structs_typedefs_enums.cbp)
...respectively:
https://svn.berlios.de/svnroot/repos/codeblocks/trunk/src/plugins/codecompletion/testing/structs_typedefs_enums.cpp (https://svn.berlios.de/svnroot/repos/codeblocks/trunk/src/plugins/codecompletion/testing/structs_typedefs_enums.cpp)
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: Martin K. on January 22, 2013, 11:01:54 am
Hi,

whats about typedef union and unnamed members? It seems that something like this:

typedef union
{
    unsigned short Value;

    struct
    {
        unsigned short Var1:6;
        unsigned short Var2:10;
    };
}
SOME_ID, *PSOME_ID;

could not be found.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: MortenMacFly on January 22, 2013, 11:36:43 am
whats about typedef union and unnamed members? It seems that something like this:
...added another test case to SVN (and fixed some warnings).
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: ollydbg on January 22, 2013, 02:05:04 pm
Hi,

whats about typedef union and unnamed members? It seems that something like this:

typedef union
{
    unsigned short Value;

    struct
    {
        unsigned short Var1:6;
        unsigned short Var2:10;
    };
}
SOME_ID, *PSOME_ID;

could not be found.

I think this bug should be discussed in another thread :), and let's focus on handling emun declaration in this thread.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: ollydbg on January 22, 2013, 02:25:38 pm
How about this:
Code
Index: src/plugins/codecompletion/parser/parserthread.cpp
===================================================================
--- src/plugins/codecompletion/parser/parserthread.cpp (wersja 8785)
+++ src/plugins/codecompletion/parser/parserthread.cpp (kopia robocza)
@@ -2263,10 +2282,13 @@
     // enums have the following rough definition:
     // enum [xxx] { type1 name1 [= 1][, [type2 name2 [= 2]]] };
     bool isUnnamed = false;
+    bool isEnumClass = false;
     int lineNr = m_Tokenizer.GetLineNumber();
     wxString token = m_Tokenizer.GetToken();
-    if (token == ParserConsts::kw_class)
+    if (token == ParserConsts::kw_class){
         token = m_Tokenizer.GetToken();
+        isEnumClass = true;
+    }
     if (token.IsEmpty())
         return;
 
@@ -2357,7 +2379,8 @@
             {
                 Token* lastParent = m_LastParent;
                 m_LastParent = newEnum;
-                DoAddToken(tkEnumerator, token, m_Tokenizer.GetLineNumber());
+                Token* enumerator = DoAddToken(tkEnumerator, token, m_Tokenizer.GetLineNumber());
+                enumerator->m_Scope = isEnumClass ? tsPrivate : tsPublic;
                 m_LastParent = lastParent;
             }
             if (peek==ParserConsts::colon)
Index: src/plugins/codecompletion/nativeparser_base.h
===================================================================
--- src/plugins/codecompletion/nativeparser_base.h (wersja 8785)
+++ src/plugins/codecompletion/nativeparser_base.h (kopia robocza)
@@ -396,14 +396,16 @@
     // for GenerateResultSet()
     bool AddChildrenOfUnnamed(TokenTree* tree, const Token* parent, TokenIdxSet& result)
     {
-        if (parent->m_TokenKind == tkClass && parent->m_Name.StartsWith(g_UnnamedSymbol))
+        if (((parent->m_TokenKind & (tkClass | tkEnum)) != 0)
+            && parent->m_Name.StartsWith(g_UnnamedSymbol))
         {
             // add all its children
             for (TokenIdxSet::const_iterator it = parent->m_Children.begin();
                                              it != parent->m_Children.end(); ++it)
             {
                 Token* tokenChild = tree->at(*it);
-                if (tokenChild)
+                if (tokenChild &&
+                    (parent->m_TokenKind == tkClass || tokenChild->m_Scope != tsPrivate))
                     result.insert(*it);
             }
 
@@ -411,7 +413,25 @@
         }
         return false;
     }
+   
+    bool AddChildrenOfEnum(TokenTree* tree, const Token* parent, TokenIdxSet& result)
+    {
+        if (parent->m_TokenKind == tkEnum)
+        {
+            // add all its children
+            for (TokenIdxSet::const_iterator it = parent->m_Children.begin();
+                                             it != parent->m_Children.end(); ++it)
+            {
+                Token* tokenChild = tree->at(*it);
+                if (tokenChild && tokenChild->m_Scope != tsPrivate)
+                    result.insert(*it);
+            }
 
+            return true;
+        }
+        return false;
+    }
+
     // for GenerateResultSet()
     bool MatchText(const wxString& text, const wxString& target, bool caseSens, bool isPrefix)
     {
Index: src/plugins/codecompletion/nativeparser_base.cpp
===================================================================
--- src/plugins/codecompletion/nativeparser_base.cpp (wersja 8785)
+++ src/plugins/codecompletion/nativeparser_base.cpp (kopia robocza)
@@ -1318,7 +1318,11 @@
                 if (!token)
                     continue;
                 if ( !AddChildrenOfUnnamed(tree, token, result) )
+                {
                     result.insert(*it);
+                    AddChildrenOfEnum(tree, token, result);
+                }
+                   
             }
 
             tree->RecalcInheritanceChain(parent);
@@ -1334,7 +1338,10 @@
                     if (!token)
                         continue;
                     if ( !AddChildrenOfUnnamed(tree, token, result) )
+                    {
                         result.insert(*it2);
+                        AddChildrenOfEnum(tree, token, result);
+                    }
                 }
             }
         }


Here is my test result of this patch:
Using the test file: trunk\src\plugins\codecompletion\testing\structs_typedefs_enums.cpp (rev 8811)

Code
//    SEnum:: OK
//    SEnum::ELocal:: OK
//    SEnum::eEnumClassLocal:: Fail
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: ollydbg on January 22, 2013, 02:43:32 pm
//    SEnum::eEnumClassLocal:: Fail
It looks like this is the expect behavior. I see the code:
Code
    while (IS_ALIVE)
    {
        // process enumerators
        token = m_Tokenizer.GetToken();
        wxString peek = m_Tokenizer.PeekToken();
        if (token.IsEmpty() || peek.IsEmpty())
            return; //eof
        if (token==ParserConsts::clbrace && level == m_Tokenizer.GetNestingLevel())
            break;
        // assignments (=xxx) are ignored by the tokenizer,
        // so we don't have to worry about them here ;)
        if (peek==ParserConsts::comma || peek==ParserConsts::clbrace || peek==ParserConsts::colon)
        {
            // this "if", avoids non-valid enumerators
            // like a comma (if no enumerators follow)
            if (   wxIsalpha(token.GetChar(0))
                || (token.GetChar(0) == ParserConsts::underscore_chr) )
            {
                Token* lastParent = m_LastParent;
                m_LastParent = newEnum;
                Token* enumerator = DoAddToken(tkEnumerator, token, m_Tokenizer.GetLineNumber());
                enumerator->m_Scope = isEnumClass ? tsPrivate : tsPublic;
                m_LastParent = lastParent;
            }
            if (peek==ParserConsts::colon)
            {
                // bit specifier (eg, xxx:1)
                // -> walk to , or }
                SkipToOneOfChars(ParserConsts::commaclbrace);
            }
        }
    }

Here:
Code
enumerator->m_Scope = isEnumClass ? tsPrivate : tsPublic;

The added enumerator is "private".

But it looks like the private members are not added to codecompletion suggestion list.
Code
    bool AddChildrenOfEnum(TokenTree* tree, const Token* parent, TokenIdxSet& result)
    {
        if (parent->m_TokenKind == tkEnum)
        {
            // add all its children
            for (TokenIdxSet::const_iterator it = parent->m_Children.begin();
                                             it != parent->m_Children.end(); ++it)
            {
                Token* tokenChild = tree->at(*it);
                if (tokenChild && tokenChild->m_Scope != tsPrivate)
                    result.insert(*it);
            }

            return true;
        }
        return false;
    }

Here, only members those are "Not private" are added to the result.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: p2rkw on January 22, 2013, 02:47:22 pm
@ollydbg:
replace
Code
SEnum::eEnumClassLocal::
with
Code
SEnum::EEnumClassLocal::
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: ollydbg on January 22, 2013, 02:50:52 pm
@ollydbg:
replace
Code
SEnum::eEnumClassLocal::
with
Code
SEnum::EEnumClassLocal::
Yes, I also see this typo, after change this, I see it list three members.
But I'm using the code:
Code
    bool AddChildrenOfEnum(TokenTree* tree, const Token* parent, TokenIdxSet& result)
    {
        if (parent->m_TokenKind == tkEnum)
        {
            // add all its children
            for (TokenIdxSet::const_iterator it = parent->m_Children.begin();
                                             it != parent->m_Children.end(); ++it)
            {
                Token* tokenChild = tree->at(*it);
                if (tokenChild )//&& tokenChild->m_Scope != tsPrivate)
                    result.insert(*it);
            }

            return true;
        }
        return false;
    }

I will test your patch again.

EDIT: your patch works fine, all three tests were OK.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: MortenMacFly on January 22, 2013, 09:25:07 pm
EDIT: your patch works fine, all three tests were OK.
Keep in mind that there are way more enums defined in this test file than only 3. These 3 commented cases are just for convenience (and because I was too lazy to type them all). But all enums should pass.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: oBFusCATed on January 29, 2013, 10:33:42 pm
Any problems encountered? If no, I'll commit in the next couple of days.
Title: Re: Patch 3407: correctly parse c++98 enums
Post by: ollydbg on January 31, 2013, 04:20:51 am
Any problems encountered? If no, I'll commit in the next couple of days.
I'm OK with this patch. All enum testes work fine
Code
//    EGlobalCommented::
//    EGlobal::
//    EGlobalAssign::
//    EEnumClassGlobal::
//    SEnum::
//    SEnum::ELocal::
//    SEnum::EEnumClassLocal::