Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => Topic started by: Alpha on October 29, 2011, 08:41:33 pm

Title: CC scope
Post by: Alpha on October 29, 2011, 08:41:33 pm

Does CC take scope into account?  If so, the code

Code

int main()
{
    {
        int fooBar = 0;
    }
    return foo //CC suggests "fooBar" here
}

says otherwise...

Title: Re: CC scope
Post by: ollydbg on October 30, 2011, 02:37:45 am

As you said, CC does not handle the "scope" information. So, when you leave the

Code

    {
        int fooBar = 0;
    }

The variable fooBar is still recognized as a local variable in main().

I think it is not easy to take scope into account.

Any ideas?

Title: Re: CC scope
Post by: Alpha on October 30, 2011, 03:07:40 am

One idea would be that when CC collects local variables, it stores them as a series of nodes (like xml).  The root node of a function would contain the local variables that are always available.  Child nodes would be created for each { } pair containing the variables declared there (child nodes would, of course, have their own child nodes for { } pairs within them).

When CC is called, it would start by looking for all the variables in the current child node, then traverse up, collecting the tokens in each parent node until it reaches the root node.

I think the hard part with this method would how to keep track of which child node the user is currently typing in.

(I am still working on reading Code::Blocks' source, so I do not yet know enough to be able to tell if implementing this method would be feasible.)

Title: Re: CC scope
Post by: ollydbg on October 30, 2011, 03:21:14 am

Quote from: Alpha on October 30, 2011, 03:07:40 am

One idea would be that when CC collects local variables, it stores them as a series of nodes (like xml).  The root node of a function would contain the local variables that are always available.  Child nodes would be created for each { } pair containing the variables declared there (child nodes would, of course, have their own child nodes for { } pairs within them).

When CC is called, it would start by looking for all the variables in the current child node, then traverse up, collecting the tokens in each parent node until it reaches the root node.

I think the hard part with this method would how to keep track of which child node the user is currently typing in.

(I am still working on reading Code::Blocks' source, so I do not yet know enough to be able to tell if implementing this method would be feasible.)

Yes, this is the correct way I think.
1, local variables were collected only when we are in a function body, we do not collect them when we do a batch parsing(batch parsing only collect declaration and skip all the function body)

2, we need a data structure to handle such scope information.

3, If you read CodeCompletion's source code, you can have a look at: Code Completion Design - CodeBlocks (http://wiki.codeblocks.org/index.php?title=Code_Completion_Design)

4, If you have any question in CC's source code, you can ask me.

Title: Re: CC scope
Post by: eranif on October 30, 2011, 07:15:43 am

Quote

I think the hard part with this method would how to keep track of which child node the user is currently typing in.

Its pretty simple actually. Assuming you know the location of the current function's body you should collect locals and manage them in a stack.

A simple container like:

Code

typedef std::map<wxString, TokenInfo> ScopedLocals; // TokenInfo represents here a class of your choice that holds into about a single local variable
typedef std::list<ScopedLocals> LocalsStack;

And then apply the following logic:

Code

start:
    stack.push_back( new ScopedLocals );

if found '{':
    stack.push_back( new ScopedLocals );

if found '}':
    delete stack.back(); // No longer relevant, free the allocated memory
    stack.pop_back();    // Remove it from the stack

if found local:
    stack.back()->insert(std::make_pair<wxString, TokenInfo>(token.Name(), token);

When you hit the end of your parsing, whatever left in the stack are your visible locals.

Special exceptions that should be handled:

- Variable defined inside 'catch()'
- Variable defined inside 'for(..)'

Eran

Title: Re: CC scope
Post by: ollydbg on October 30, 2011, 03:26:19 pm

@eranif (http://forums.codeblocks.org/index.php?action=profile;u=1055)
Dear eranif, thanks for your direction. I guess that this has done in your Codelite IDE, it is in your Bison(yacc) based parser, right?
Yes, it can be implemented, I'm thinking how this mechanism can be added to our cc's parser. (c::b's parser should be extend to handle brace scope information, we don't have a separate parser to parse the function body, so it may be add code everywhere in the recent parser's code)

Title: Re: CC scope
Post by: Alpha on October 30, 2011, 06:36:33 pm

Quote from: eranif on October 30, 2011, 07:15:43 am

Quote

I think the hard part with this method would how to keep track of which child node the user is currently typing in.

Its pretty simple actually. Assuming you know the location of the current function's body you should collect locals and manage them in a stack.

That makes sense (unless I have completely misunderstood you :)); I had been thinking along the lines of storing tokens, not re-parsing on cursor movements (except if made necessary by the user's edits).
(As you can tell, I still have a lot to learn about Code::Blocks' inner workings.)

Title: Re: CC scope
Post by: eranif on October 30, 2011, 06:57:22 pm

Quote

it is in your Bison(yacc) based parser, right?

Actually, I don't parse locals until they are needed (e.g. user is trying to code complete something).

I wrote a lexer for this purpose (bison is not really needed here, I managed to do it with flex only)
http://codelite.svn.sourceforge.net/viewvc/codelite/trunk/ScopeOptimizer/ScopeOptimizer/scope_optimizer.l?revision=5183&view=markup (http://codelite.svn.sourceforge.net/viewvc/codelite/trunk/ScopeOptimizer/ScopeOptimizer/scope_optimizer.l?revision=5183&view=markup)

What this lexer does, its actually "optimize the scope", that means that it will remove all hidden blocks so the parser won't have to deal with them,
so this:

Code

int main(int argc,char **argv) {
 {
   int member;
 }
  int another_variable;
| // caret is here

will become this:

Code

int main(int argc,char **argv) {
 {}
  int another_variable;

note that this code is very independent (it comes with its own codelite workspace) and it expose a very simple API:

Code

int OptimizeScope(const std::string &inputScope, std::string &optimizedScope)

The first argument is the buffer you want to optimize while the later is the result (the output).

Eran

Title: Re: CC scope
Post by: ollydbg on October 31, 2011, 07:34:30 am

Quote from: eranif on October 30, 2011, 06:57:22 pm

Quote

it is in your Bison(yacc) based parser, right?

Actually, I don't parse locals until they are needed (e.g. user is trying to code complete something).

Yes, Codeblocks' CC do the same thing as it needed(e.g. when user select context menu->find declaration of XXXXX).

Quote

I wrote a lexer for this purpose (bison is not really needed here, I managed to do it with flex only)
http://codelite.svn.sourceforge.net/viewvc/codelite/trunk/ScopeOptimizer/ScopeOptimizer/scope_optimizer.l?revision=5183&view=markup (http://codelite.svn.sourceforge.net/viewvc/codelite/trunk/ScopeOptimizer/ScopeOptimizer/scope_optimizer.l?revision=5183&view=markup)

What this lexer does, its actually "optimize the scope", that means that it will remove all hidden blocks so the parser won't have to deal with them,
so this:

Code

int main(int argc,char **argv) {
 {
   int member;
 }
  int another_variable;
| // caret is here

will become this:

Code

int main(int argc,char **argv) {
 {}
  int another_variable;

note that this code is very independent (it comes with its own codelite workspace) and it expose a very simple API:

Code

int OptimizeScope(const std::string &inputScope, std::string &optimizedScope)

The first argument is the buffer you want to optimize while the later is the result (the output).
Eran

Codeblocks' current implementation lacks such feature, In the currently CC, we just let the parser parse the function body(we say: buffer) until the caret. So, as your said, I think c::b's cc can improve this by modify the buffer, just like you mentioned. The, we can let our parser works in the modified buffer(optimized scope string as in your example).

That's really cool! thanks for the hint, so the last thing is to take some time to implement it. :D

Title: Re: CC scope
Post by: Alpha on November 17, 2012, 11:21:28 pm

Here is a possible scope optimizer for CC.  If anyone has time to test, let me know how well it works.

Code

Index: src/plugins/codecompletion/nativeparser.cpp
===================================================================
--- src/plugins/codecompletion/nativeparser.cpp	(revision 8571)
+++ src/plugins/codecompletion/nativeparser.cpp	(working copy)
@@ -1826,7 +1826,7 @@
     if (blockStart != -1)
     {
         ++blockStart; // skip {
-        const int pos = caretPos == -1 ? searchData->control->GetCurrentPos() : caretPos;
+        const int pos = (caretPos == -1 ? searchData->control->GetCurrentPos() : caretPos);
         const int line = searchData->control->LineFromPosition(pos);
         const int blockEnd = searchData->control->GetLineEndPosition(line);
         if (blockEnd < 0 || blockEnd > searchData->control->GetLength())
@@ -1842,7 +1842,29 @@
         if (blockStart >= blockEnd)
             blockStart = blockEnd;
 
-        wxString buffer = searchData->control->GetTextRange(blockStart, blockEnd);
+        wxString buffer; // = searchData->control->GetTextRange(blockStart, blockEnd);
+        // condense out-of-scope braces {...}
+        int scanPos = blockEnd;
+        for (int i = pos; i > blockStart; --i)
+        {
+            if (searchData->control->GetCharAt(i) != wxT('}'))
+                continue;
+            const int style = searchData->control->GetStyleAt(i);
+            if (   searchData->control->IsString(style)
+                || searchData->control->IsCharacter(style)
+                || searchData->control->IsComment(style))
+            {
+                continue;
+            }
+            const int scopeStart = searchData->control->BraceMatch(i);
+            if (scopeStart < blockStart)
+                break;
+            buffer.Prepend(searchData->control->GetTextRange(i, scanPos));
+            scanPos = scopeStart + 1;
+            i       = scopeStart;
+        }
+        buffer.Prepend(searchData->control->GetTextRange(blockStart, scanPos));
+
         buffer.Trim();
         if (   !buffer.IsEmpty()
             && !m_Parser->ParseBuffer(buffer, false, false, true, searchData->file, m_LastFuncTokenIdx, initLine) )

Title: Re: CC scope
Post by: ollydbg on November 19, 2012, 06:36:12 am

Quote from: Alpha on November 17, 2012, 11:21:28 pm

Here is a possible scope optimizer for CC.  If anyone has time to test, let me know how well it works.

Code

Index: src/plugins/codecompletion/nativeparser.cpp
===================================================================
--- src/plugins/codecompletion/nativeparser.cpp	(revision 8571)
+++ src/plugins/codecompletion/nativeparser.cpp	(working copy)
@@ -1826,7 +1826,7 @@
     if (blockStart != -1)
     {
         ++blockStart; // skip {
-        const int pos = caretPos == -1 ? searchData->control->GetCurrentPos() : caretPos;
+        const int pos = (caretPos == -1 ? searchData->control->GetCurrentPos() : caretPos);
         const int line = searchData->control->LineFromPosition(pos);
         const int blockEnd = searchData->control->GetLineEndPosition(line);
         if (blockEnd < 0 || blockEnd > searchData->control->GetLength())
@@ -1842,7 +1842,29 @@
         if (blockStart >= blockEnd)
             blockStart = blockEnd;
 
-        wxString buffer = searchData->control->GetTextRange(blockStart, blockEnd);
+        wxString buffer; // = searchData->control->GetTextRange(blockStart, blockEnd);
+        // condense out-of-scope braces {...}
+        int scanPos = blockEnd;
+        for (int i = pos; i > blockStart; --i)
+        {
+            if (searchData->control->GetCharAt(i) != wxT('}'))
+                continue;
+            const int style = searchData->control->GetStyleAt(i);
+            if (   searchData->control->IsString(style)
+                || searchData->control->IsCharacter(style)
+                || searchData->control->IsComment(style))
+            {
+                continue;
+            }
+            const int scopeStart = searchData->control->BraceMatch(i);
+            if (scopeStart < blockStart)
+                break;
+            buffer.Prepend(searchData->control->GetTextRange(i, scanPos));
+            scanPos = scopeStart + 1;
+            i       = scopeStart;
+        }
+        buffer.Prepend(searchData->control->GetTextRange(blockStart, scanPos));
+
         buffer.Trim();
         if (   !buffer.IsEmpty()
             && !m_Parser->ParseBuffer(buffer, false, false, true, searchData->file, m_LastFuncTokenIdx, initLine) )

The logic seems good to me.

So, for a simple code:

Code

void ffff()
{
    int a;
    {
        int b;
        {
            int c;
        }
    }
    
    for (....)
    {
        int d;      
        {
            int e;
            ////////// caret here;       
        }
    }
}

It should have only variable statement "e, d, a" in the buffer.

I'm going to test it in tonight. :)

Title: Re: CC scope
Post by: ollydbg on November 19, 2012, 11:39:54 am

Two issue:
1, if you remove the whole "{" block, you will remove all the EOL characters "\n", so after parsing the buffer, you variable location (line information) may be changed.

2, if you remove the "{" block directly, which means:

Code

for (...)
{
     int a;
}

will become

Code

for (...)

This becomes a invalid statement. What I think is right method is to convert to this:

Code

for (...)
{
     
}

Keep the "{" and "}" and all the EOL characters, and remove all others. :)

Title: Re: CC scope
Post by: MortenMacFly on November 19, 2012, 12:13:57 pm

Quote from: ollydbg on November 19, 2012, 11:39:54 am

Code

for (...)

This becomes a invalid statement. What I think is right method is to convert to this:

Code

for (...)
{
}

True. Otherwise our parser may get hick-ups. ;-)

Also, (ollydbg): Do you have applied that other patch concerning variable parsing of for loops and stuff in parallel? This could conflict.

Title: Re: CC scope
Post by: ollydbg on November 19, 2012, 02:09:49 pm

Quote from: MortenMacFly on November 19, 2012, 12:13:57 pm

Also, (ollydbg): Do you have applied that other patch concerning variable parsing of for loops and stuff in parallel? This could conflict.

Yes, this patch did conflict with the one for parsing of for loops. But sorry those days I'm a little busy, so I have no time to deeply test and merge those two patches. :(

Title: Re: CC scope
Post by: MortenMacFly on November 19, 2012, 02:43:25 pm

Quote from: ollydbg on November 19, 2012, 02:09:49 pm

Yes, this patch did conflict with the one for parsing of for loops. But sorry those days I'm a little busy, so I have no time to deeply test and merge those two patches. :(

...maybe ALPHA can do it - what I meant is this:
http://developer.berlios.de/patch/?func=detailpatch&patch_id=3345&group_id=5358

Title: Re: CC scope
Post by: Alpha on November 19, 2012, 11:08:27 pm

Quote from: ollydbg on November 19, 2012, 11:39:54 am

1, if you remove the whole "{" block, you will remove all the EOL characters "\n", so after parsing the buffer, you variable location (line information) may be changed.

I never thought of that... I will see what I can do.

Quote from: ollydbg on November 19, 2012, 11:39:54 am

2, if you remove the "{" block directly, which means:

The posted patch already deals with this.

Code

+            buffer.Prepend(searchData->control->GetTextRange(i, scanPos));
+            scanPos = scopeStart + 1;

i is the position of a "}" brace, and the + 1 includes the matching "{" brace in scanPos, so

Code

for (...)
{
     int a;
}

currently becomes

Code

for (...)
{}

Quote from: MortenMacFly on November 19, 2012, 02:43:25 pm

Quote from: ollydbg on November 19, 2012, 02:09:49 pm

Yes, this patch did conflict with the one for parsing of for loops. But sorry those days I'm a little busy, so I have no time to deeply test and merge those two patches. :(

...maybe ALPHA can do it - what I meant is this:
http://developer.berlios.de/patch/?func=detailpatch&patch_id=3345&group_id=5358

Have not looked at that patch yet, but if it is not too complicated, I will merge them.

Title: Re: CC scope
Post by: Alpha on November 20, 2012, 11:36:19 pm

Quote from: Alpha on November 19, 2012, 11:08:27 pm

Quote from: ollydbg on November 19, 2012, 11:39:54 am

1, if you remove the whole "{" block, you will remove all the EOL characters "\n", so after parsing the buffer, you variable location (line information) may be changed.

I never thought of that... I will see what I can do.

This should deal with it:

Code

Index: src/plugins/codecompletion/nativeparser.cpp
===================================================================
--- src/plugins/codecompletion/nativeparser.cpp	(revision 8585)
+++ src/plugins/codecompletion/nativeparser.cpp	(working copy)
@@ -1857,7 +1857,33 @@
         if (blockStart >= blockEnd)
             blockStart = blockEnd;
 
-        wxString buffer = searchData->control->GetTextRange(blockStart, blockEnd);
+        wxString buffer; // = searchData->control->GetTextRange(blockStart, blockEnd);
+        // condense out-of-scope braces {...}
+        int scanPos = blockEnd;
+        for (int curPos = pos; curPos > blockStart; --curPos)
+        {
+            if (searchData->control->GetCharAt(curPos) != wxT('}'))
+                continue;
+            const int style = searchData->control->GetStyleAt(curPos);
+            if (   searchData->control->IsString(style)
+                || searchData->control->IsCharacter(style)
+                || searchData->control->IsComment(style))
+            {
+                continue;
+            }
+            const int scopeStart = searchData->control->BraceMatch(curPos);
+            if (scopeStart < blockStart)
+                break;
+            buffer.Prepend(searchData->control->GetTextRange(curPos, scanPos));
+            const int startLn = searchData->control->LineFromPosition(scopeStart);
+            const int endLn   = searchData->control->LineFromPosition(curPos);
+            if (startLn < endLn) // maintain correct line numbers for parsed tokens
+                buffer.Prepend( wxString(wxT('\n'), endLn - startLn) );
+            scanPos = scopeStart + 1;
+            curPos  = scopeStart;
+        }
+        buffer.Prepend(searchData->control->GetTextRange(blockStart, scanPos));
+
         buffer.Trim();
         if (   !buffer.IsEmpty()
             && !m_Parser->ParseBuffer(buffer, false, false, true, searchData->file, m_LastFuncTokenIdx, initLine) )

Quote from: Alpha on November 19, 2012, 11:08:27 pm

Quote from: MortenMacFly on November 19, 2012, 02:43:25 pm

Quote from: ollydbg on November 19, 2012, 02:09:49 pm

Yes, this patch did conflict with the one for parsing of for loops. But sorry those days I'm a little busy, so I have no time to deeply test and merge those two patches. :(

...maybe ALPHA can do it - what I meant is this:
http://developer.berlios.de/patch/?func=detailpatch&patch_id=3345&group_id=5358

Have not looked at that patch yet, but if it is not too complicated, I will merge them.

The only conflict I can see (are there other conflicts I have missed?) is that:

Code

int main()
{
    for (int var_it = 0; var_it < 5; ++var_it)
    {
        int fooBar = var_it;
    }
    // "fooBar" correctly does not exist here (with the above patch)
    // but "var_it" does exist (from loop declaration patch)
    return 0;
}

When modifying buffer, I think the proper way to condense for loop arguments is:

Code

for (int var_it = 0; var_it < 5; ++var_it)
{
    // stuff...
}

to:

Code

for (;;)
{

}

However, I am not certain about the other cases that the loop declaration patch handles; any suggestions?

Title: Re: CC scope
Post by: Alpha on November 22, 2012, 04:48:57 am

Submitted patch 3375 (http://developer.berlios.de/patch/index.php?func=detailpatch&patch_id=3375&group_id=5358).  It should address all issues discussed here.