Author Topic: make cc real-time parse  (Read 19127 times)

Offline blueshake

  • Regular
  • ***
  • Posts: 459
make cc real-time parse
« on: December 04, 2009, 02:32:48 pm »
hi,

this patch can make cc parse in real-time ,you do not need to save file to force the parser to parse the file now.
but
as I post in this thread,http://forums.codeblocks.org/index.php/topic,11599.msg78916.html#msg78916

there are some problems with NullLoader calling. :wink:

comment is welcome.

« Last Edit: December 06, 2009, 11:23:14 am by blueshake »
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: make cc real-tiem parse
« Reply #1 on: December 04, 2009, 03:24:13 pm »
Very nice work!!!

Your post title should be "real-time parse", there is a typo. :D
I will test it as soon as possible.

By the way, I have just passed the oral defense of PHD thesis today :wink:, I will have a travel tomorrow.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ironhead

  • Almost regular
  • **
  • Posts: 210
Re: make cc real-tiem parse
« Reply #2 on: December 04, 2009, 07:45:00 pm »
Won't this continually drive CPU usage as the user types given that you are forcing a reparse on the  wxSCI_MOD_INSERTTEXT and wxSCI_MOD_DELETETEXT events?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-tiem parse
« Reply #3 on: December 05, 2009, 11:48:09 am »
Won't this continually drive CPU usage as the user types given that you are forcing a reparse on the  wxSCI_MOD_INSERTTEXT and wxSCI_MOD_DELETETEXT events?


on the event above and in different line will reparse the file.

Actually it is the first step,in the next move I will optimize the cc search based on this.
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-time parse
« Reply #4 on: December 06, 2009, 02:41:46 am »
um, it will crash if there are files which were not colsed last time in the filemanager.
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-time parse
« Reply #5 on: December 06, 2009, 02:47:43 am »
can try this and thanks thomas for his advice in this threadhttp://forums.codeblocks.org/index.php/topic,11599.msg79077.html#msg79077

« Last Edit: December 06, 2009, 11:22:49 am by blueshake »
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-time parse
« Reply #6 on: December 06, 2009, 11:22:23 am »
change the inner copy method in EditorReuser,see my post in this thread.http://forums.codeblocks.org/index.php/topic,11599.msg79121.html#msg79121


patch:


Code
Index: src/include/filemanager.h
===================================================================
--- src/include/filemanager.h (revision 5951)
+++ src/include/filemanager.h (working copy)
@@ -144,7 +144,19 @@
     NullLoader(const wxString& name, char* buffer, size_t size) { fileName = name; data = buffer; len = size; Ready(); };
     void operator()(){};
 };
-
+class EditorReuser : public LoaderBase
+{
+public:
+    EditorReuser(const wxString& name, const wxString& s)
+    {
+        fileName = name;
+        len = strlen(s.mb_str(wxConvUTF8));
+        data = new char[len + 1];
+        strcpy(data, (const char*)s.mb_str(wxConvUTF8));
+        Ready();
+    }
+    void operator()(){};
+};
 // ***** class: FileManager *****
 class FileManager : public Mgr<FileManager>
 {
Index: src/plugins/codecompletion/codecompletion.cpp
===================================================================
--- src/plugins/codecompletion/codecompletion.cpp (revision 5951)
+++ src/plugins/codecompletion/codecompletion.cpp (working copy)
@@ -172,7 +172,8 @@
     m_IsAutoPopup(false),
     m_ToolbarChanged(true),
     m_CurrentLine(0),
-    m_LastFile(wxEmptyString)
+    m_LastFile(wxEmptyString),
+    m_NeedReparse(false)
 {
     if(!Manager::LoadResource(_T("codecompletion.zip")))
     {
@@ -2069,7 +2070,11 @@
             }
         }
     }
-
+    if ((event.GetModificationType() & wxSCI_MOD_INSERTTEXT) ||
+        (event.GetModificationType() & wxSCI_MOD_DELETETEXT))
+    {
+        m_NeedReparse = true;
+    }
     if( control->GetCurrentLine() != m_CurrentLine)
     {
         m_CurrentLine = control->GetCurrentLine();
@@ -2093,6 +2098,16 @@
                 m_Scope->SetSelection(wxNOT_FOUND);
             }
         }
+        if (m_NeedReparse)
+        {
+            Parser* parser = m_NativeParser.FindParserFromActiveEditor();
+            if (parser)
+            {
+                parser->Reparse(editor->GetFilename());
+            }
+            m_NeedReparse= false;
+        }
+
     }
 
     // allow others to handle this event
Index: src/plugins/codecompletion/codecompletion.h
===================================================================
--- src/plugins/codecompletion/codecompletion.h (revision 5951)
+++ src/plugins/codecompletion/codecompletion.h (working copy)
@@ -146,7 +146,7 @@
         int                                m_CurrentLine;
         map<wxString, int>                 m_SearchItem;
         wxString                           m_LastFile;
-
+        bool                               m_NeedReparse;
         bool                               m_LexerKeywordsToInclude[9];
 
         DECLARE_EVENT_TABLE()
Index: src/plugins/codecompletion/parser/parser.cpp
===================================================================
--- src/plugins/codecompletion/parser/parser.cpp (revision 5951)
+++ src/plugins/codecompletion/parser/parser.cpp (working copy)
@@ -538,7 +538,7 @@
             }
 
             if (!opts.loader) //this should always be true (memory will leak if a loader has already been initialized before this point)
-                opts.loader = Manager::Get()->GetFileManager()->Load(bufferOrFilename, false);
+                opts.loader = Manager::Get()->GetFileManager()->Load(bufferOrFilename, true);
         }
 
 #if PARSER_DEBUG_OUTPUT
Index: src/sdk/filemanager.cpp
===================================================================
--- src/sdk/filemanager.cpp (revision 5951)
+++ src/sdk/filemanager.cpp (working copy)
@@ -138,12 +138,8 @@
                 cbEditor* ed = em->GetBuiltinEditor(em->GetEditor(i));
                 if(ed && fileName == ed->GetFilename())
                 {
-                    wxString s(ed->GetControl()->GetText());
-                    #if wxCHECK_VERSION(2, 9, 0)
-                    NullLoader *nl = new NullLoader(file, (char*) s.wx_str(), s.length() * sizeof(wxChar));
-                    #else
-                    NullLoader *nl = new NullLoader(file, (char*) s.c_str(), s.length() * sizeof(wxChar));
-                    #endif
+
+                    EditorReuser *nl = new EditorReuser(file, ed->GetControl()->GetText());
                     return nl;
                 }
             }
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-time parse
« Reply #7 on: December 14, 2009, 01:40:41 am »
optimize the cc search process based on real-time parse.

before:


parse the contexts between the begin and the caret position to get the function tokens.


now

get the function tokens from the tokentree with filename directly based on real-time parse.and this reduce the parse time.

enjoy it. :D


patch:

Code
Index: src/include/filemanager.h
===================================================================
--- src/include/filemanager.h (revision 5973)
+++ src/include/filemanager.h (working copy)
@@ -144,7 +144,19 @@
     NullLoader(const wxString& name, char* buffer, size_t size) { fileName = name; data = buffer; len = size; Ready(); };
     void operator()(){};
 };
-
+class EditorReuser : public LoaderBase
+{
+public:
+    EditorReuser(const wxString& name, const wxString& s)
+    {
+        fileName = name;
+        len = strlen(s.mb_str(wxConvUTF8));
+        data = new char[len + 1];
+        strcpy(data, (const char*)s.mb_str(wxConvUTF8));
+        Ready();
+    }
+    void operator()(){};
+};
 // ***** class: FileManager *****
 class FileManager : public Mgr<FileManager>
 {
Index: src/plugins/codecompletion/codecompletion.cpp
===================================================================
--- src/plugins/codecompletion/codecompletion.cpp (revision 5973)
+++ src/plugins/codecompletion/codecompletion.cpp (working copy)
@@ -172,7 +172,8 @@
     m_IsAutoPopup(false),
     m_ToolbarChanged(true),
     m_CurrentLine(0),
-    m_LastFile(wxEmptyString)
+    m_LastFile(wxEmptyString),
+    m_NeedReparse(false)
 {
     if(!Manager::LoadResource(_T("codecompletion.zip")))
     {
@@ -2069,7 +2070,11 @@
             }
         }
     }
-
+    if ((event.GetModificationType() & wxSCI_MOD_INSERTTEXT) ||
+        (event.GetModificationType() & wxSCI_MOD_DELETETEXT))
+    {
+        m_NeedReparse = true;
+    }
     if( control->GetCurrentLine() != m_CurrentLine)
     {
         m_CurrentLine = control->GetCurrentLine();
@@ -2093,6 +2098,16 @@
                 m_Scope->SetSelection(wxNOT_FOUND);
             }
         }
+        if (m_NeedReparse)
+        {
+            Parser* parser = m_NativeParser.FindParserFromActiveEditor();
+            if (parser)
+            {
+                parser->Reparse(editor->GetFilename());
+            }
+            m_NeedReparse= false;
+        }
+
     }
 
     // allow others to handle this event
Index: src/plugins/codecompletion/codecompletion.h
===================================================================
--- src/plugins/codecompletion/codecompletion.h (revision 5973)
+++ src/plugins/codecompletion/codecompletion.h (working copy)
@@ -146,7 +146,7 @@
         int                                m_CurrentLine;
         map<wxString, int>                 m_SearchItem;
         wxString                           m_LastFile;
-
+        bool                               m_NeedReparse;
         bool                               m_LexerKeywordsToInclude[9];
 
         DECLARE_EVENT_TABLE()
Index: src/plugins/codecompletion/nativeparser.cpp
===================================================================
--- src/plugins/codecompletion/nativeparser.cpp (revision 5973)
+++ src/plugins/codecompletion/nativeparser.cpp (working copy)
@@ -2152,18 +2152,13 @@
     }
     s_LastEditor = editor;
     s_LastLine = line;
-
-    Parser parser(this);
-    parser.ParseBufferForFunctions(control->GetTextRange(0, pos));
-
-    wxArrayString funcs;
-    TokensTree* tmptree = parser.GetTempTokens();
-
-    // look for implementation functions that enclose our current line number
-    for(size_t i = 0; i < tmptree->size();i++)
+    TokenIdxSet result;
+    m_Parser.FindTokensInFile(editor->GetFilename(), result, tkFunction|tkConstructor|tkDestructor);
+    TokensTree* tree = m_Parser.GetTokens();
+    for (TokenIdxSet::iterator it = result.begin(); it != result.end(); ++it)
     {
-        Token* token = tmptree->at(i);
-        if (token && (token->m_TokenKind == tkFunction || token->m_TokenKind == tkConstructor || token->m_TokenKind == tkDestructor))
+        Token* token = tree->at(*it);
+        if (token)
         {
             // found a function; check its bounds
             if (token->m_ImplLineStart <= (size_t)line && token->m_ImplLineEnd >= (size_t)line)
@@ -2174,6 +2169,7 @@
                                                                 token->DisplayName().wx_str(),
                                                                 token->m_ImplLine));
 
+
                 s_LastNS = token->GetNamespace();
                 s_LastPROC = token->m_Name;
                 s_LastResult = control->PositionFromLine(token->m_ImplLine - 1);
Index: src/plugins/codecompletion/parser/parser.cpp
===================================================================
--- src/plugins/codecompletion/parser/parser.cpp (revision 5973)
+++ src/plugins/codecompletion/parser/parser.cpp (working copy)
@@ -538,7 +538,7 @@
             }
 
             if (!opts.loader) //this should always be true (memory will leak if a loader has already been initialized before this point)
-                opts.loader = Manager::Get()->GetFileManager()->Load(bufferOrFilename, false);
+                opts.loader = Manager::Get()->GetFileManager()->Load(bufferOrFilename, true);
         }
 
 #if PARSER_DEBUG_OUTPUT
@@ -1045,3 +1045,22 @@
     }
     return true;
 }
+size_t Parser::FindTokensInFile(const wxString& fileName,TokenIdxSet& result,short int kindMask)
+{
+    result.clear();
+    wxString file = UnixFilename(fileName);
+    TokenIdxSet tmpresult;
+    wxCriticalSectionLocker lock(s_MutexProtection);
+    if(!m_pTokens->FindTokensInFile(file,tmpresult,kindMask))
+        return 0;
+
+    TokenIdxSet::iterator it;
+    for(it = tmpresult.begin();it!=tmpresult.end();++it)
+    {
+        Token* token = m_pTokens->at(*it);
+        if(token)
+        //result.push_back(token);
+        result.insert(*it);
+    }
+    return result.size();
+}
Index: src/plugins/codecompletion/parser/parser.h
===================================================================
--- src/plugins/codecompletion/parser/parser.h (revision 5973)
+++ src/plugins/codecompletion/parser/parser.h (working copy)
@@ -153,6 +153,7 @@
         Token* FindChildTokenByName(Token* parent, const wxString& name, bool useInheritance = false, short int kindMask = 0xFFFF) const;
         size_t FindMatches(const wxString& s, TokenList&   result, bool caseSensitive = true, bool is_prefix = true);
         size_t FindMatches(const wxString& s, TokenIdxSet& result, bool caseSensitive = true, bool is_prefix = true);
+        size_t FindTokensInFile(const wxString& fileName, TokenIdxSet& result, short int kindMask);
         ParserOptions& Options() { return m_Options; }
         BrowserOptions& ClassBrowserOptions() { return m_BrowserOptions; }
 
Index: src/sdk/filemanager.cpp
===================================================================
--- src/sdk/filemanager.cpp (revision 5973)
+++ src/sdk/filemanager.cpp (working copy)
@@ -138,12 +138,8 @@
                 cbEditor* ed = em->GetBuiltinEditor(em->GetEditor(i));
                 if(ed && fileName == ed->GetFilename())
                 {
-                    wxString s(ed->GetControl()->GetText());
-                    #if wxCHECK_VERSION(2, 9, 0)
-                    NullLoader *nl = new NullLoader(file, (char*) s.wx_str(), s.length() * sizeof(wxChar));
-                    #else
-                    NullLoader *nl = new NullLoader(file, (char*) s.c_str(), s.length() * sizeof(wxChar));
-                    #endif
+
+                    EditorReuser *nl = new EditorReuser(file, ed->GetControl()->GetText());
                     return nl;
                 }
             }
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: make cc real-time parse
« Reply #8 on: December 14, 2009, 02:17:01 pm »
works fine!! Thanks for sharing!
@blueshake:
Please comment these codes in your patch.

Code
    if (result.size()<1) // found nothing in the search_scope, add global namespace
    {
        if (s_DebugSmartSense)
        {
            Manager::Get()->GetLogManager()->DebugLog(F(_T("AI() result is zero, so, add the Global namespace")));
        }
        search_scope->insert(-1);
        FindAIMatches(parser, components, result, -1, noPartialMatch, caseSensitive, true, 0xffff, search_scope);
    }


Because global search scope is already included, so we don't search it again. :D
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: make cc real-time parse
« Reply #9 on: December 14, 2009, 02:25:40 pm »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-time parse
« Reply #10 on: December 15, 2009, 02:48:33 am »
But this bug CC fail after A class's static method called. still exist. :(

I think it is something wrong with parsethread.
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-time parse
« Reply #11 on: December 23, 2009, 12:06:18 pm »
Another patch for improvement.

Patch:




Note:

All of these improvments are based on the real-time parse.
« Last Edit: December 25, 2009, 02:46:47 am by blueshake »
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-time parse
« Reply #12 on: December 25, 2009, 02:50:00 am »
Some modifations.


1.remove the type conditions which were done by the search.
2.adjust the update logic.

patch:


Code
Index: src/plugins/codecompletion/codecompletion.cpp
===================================================================
--- src/plugins/codecompletion/codecompletion.cpp (revision 5986)
+++ src/plugins/codecompletion/codecompletion.cpp (working copy)
@@ -1410,16 +1410,23 @@
         funcdata->m_NameSpaces.clear();
         funcdata->parsed = true;
 
-        Parser parser(this);
+        /*Parser parser(this);
         TokensTree* tmptree = parser.GetTempTokens();
         tmptree->clear();
         parser.ParseBufferForFunctions(ed->GetControl()->GetText());
 
         for(size_t i = 0; i < tmptree->size(); ++i)
+        */
+        Parser* parser = m_NativeParser.FindParserFromEditor(ed);
+        if (!parser)
+            return;
+        TokenIdxSet result;
+        TokensTree* tmptree = parser->GetTokens();
+        parser->FindTokensInFile(filename, result, tkFunction|tkConstructor|tkDestructor);
+        for (TokenIdxSet::iterator it =result.begin(); it != result.end(); ++it)
         {
-            const Token* token = tmptree->at(i);
-            if (token && (token->m_TokenKind == tkFunction || token->m_TokenKind == tkConstructor || token->m_TokenKind == tkDestructor)
-                && token->m_ImplLine != 0)
+            const Token* token = tmptree->at(*it);
+            if (token && token->m_ImplLine != 0)//&& (token->m_TokenKind == tkFunction || token->m_TokenKind == tkConstructor || token->m_TokenKind == tkDestructor)
             {
                 FunctionScope func;
                 func.StartLine = token->m_ImplLine - 1;
@@ -2077,28 +2084,6 @@
     }
     if( control->GetCurrentLine() != m_CurrentLine)
     {
-        m_CurrentLine = control->GetCurrentLine();
-        int sel = FunctionPosition();
-        if(sel != -1 && sel != m_Function->GetSelection())
-        {
-            m_Function->SetSelection(sel);
-            m_Scope->SetSelection(sel);
-        }
-        else if(sel == -1)
-        {
-            m_Function->SetSelection(wxNOT_FOUND);
-            // TO DO : set scope correctly
-            int NsSel = NameSpacePosition();
-            if(NsSel != -1)
-            {
-                m_Scope->SetSelection(NsSel + m_StartIdxNameSpaceInScope);
-            }
-            else
-            {
-                m_Scope->SetSelection(wxNOT_FOUND);
-            }
-        }
-
         if (m_NeedReparse)
         {
             Parser* parser = m_NativeParser.FindParserFromActiveEditor();
@@ -2107,7 +2092,32 @@
                 parser->Reparse(editor->GetFilename());
             }
             m_NeedReparse= false;
+            //m_CurrentLine = control->GetCurrentLine();
         }
+        else
+        {
+            m_CurrentLine = control->GetCurrentLine();
+            int sel = FunctionPosition();
+            if(sel != -1 && sel != m_Function->GetSelection())
+            {
+                m_Function->SetSelection(sel);
+                m_Scope->SetSelection(sel);
+            }
+            else if(sel == -1)
+            {
+                m_Function->SetSelection(wxNOT_FOUND);
+                // TO DO : set scope correctly
+                int NsSel = NameSpacePosition();
+                if(NsSel != -1)
+                {
+                    m_Scope->SetSelection(NsSel + m_StartIdxNameSpaceInScope);
+                }
+                else
+                {
+                    m_Scope->SetSelection(wxNOT_FOUND);
+                }
+            }
+        }
     }
 
     // allow others to handle this event
@@ -2133,4 +2143,5 @@
 void CodeCompletion::OnParserEnd(wxCommandEvent& event)
 {
     // nothing for now
+    ParseFunctionsAndFillToolbar(true);
 }
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: make cc real-time parse
« Reply #13 on: December 25, 2009, 03:10:47 am »
I will test this patch, Thanks for the contribution!!! :D
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: make cc real-time parse
« Reply #14 on: January 07, 2010, 10:10:48 am »
optimize the cc search process based on real-time parse.
[...]
Code
Index: src/include/filemanager.h
===================================================================
--- src/plugins/codecompletion/nativeparser.cpp (revision 5973)
+++ src/plugins/codecompletion/nativeparser.cpp (working copy)
@@ -2152,18 +2152,13 @@
     }
     s_LastEditor = editor;
     s_LastLine = line;
-
-    Parser parser(this);
-    parser.ParseBufferForFunctions(control->GetTextRange(0, pos));
-
-    wxArrayString funcs;
-    TokensTree* tmptree = parser.GetTempTokens();
-
-    // look for implementation functions that enclose our current line number
-    for(size_t i = 0; i < tmptree->size();i++)
+    TokenIdxSet result;
+    m_Parser.FindTokensInFile(editor->GetFilename(), result, tkFunction|tkConstructor|tkDestructor);
+    TokensTree* tree = m_Parser.GetTokens();
+    for (TokenIdxSet::iterator it = result.begin(); it != result.end(); ++it)
     {
-        Token* token = tmptree->at(i);
-        if (token && (token->m_TokenKind == tkFunction || token->m_TokenKind == tkConstructor || token->m_TokenKind == tkDestructor))
+        Token* token = tree->at(*it);
+        if (token)
         {
             // found a function; check its bounds
             if (token->m_ImplLineStart <= (size_t)line && token->m_ImplLineEnd >= (size_t)line)
@@ -2174,6 +2169,7 @@
                                                                 token->DisplayName().wx_str(),
                                                                 token->m_ImplLine));
 
+
                 s_LastNS = token->GetNamespace();
                 s_LastPROC = token->m_Name;
                 s_LastResult = control->PositionFromLine(token->m_ImplLine - 1);
This is certainly the cause for the wrong file being returned which I've reported here:
http://forums.codeblocks.org/index.php/topic,11800.msg80076.html#msg80076
Look: Previously the parser was created based on the native parsers current state. Now you just use the attached parser which maybe wrong because it may point to the wrong file!
I believe something like FindParserFromEditor needs to be used here...
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-time parse
« Reply #15 on: January 07, 2010, 12:36:39 pm »
optimize the cc search process based on real-time parse.

before:


parse the contexts between the begin and the caret position to get the function tokens.


now

get the function tokens from the tokentree with filename directly based on real-time parse.and this reduce the parse time.

enjoy it. :D


patch:

Code
Index: src/include/filemanager.h
===================================================================
--- src/include/filemanager.h (revision 5973)
+++ src/include/filemanager.h (working copy)
@@ -144,7 +144,19 @@
     NullLoader(const wxString& name, char* buffer, size_t size) { fileName = name; data = buffer; len = size; Ready(); };
     void operator()(){};
 };
-
+class EditorReuser : public LoaderBase
+{
+public:
+    EditorReuser(const wxString& name, const wxString& s)
+    {
+        fileName = name;
+        len = strlen(s.mb_str(wxConvUTF8));
+        data = new char[len + 1];
+        strcpy(data, (const char*)s.mb_str(wxConvUTF8));
+        Ready();
+    }
+    void operator()(){};
+};
 // ***** class: FileManager *****
 class FileManager : public Mgr<FileManager>
 {
Index: src/plugins/codecompletion/codecompletion.cpp
===================================================================
--- src/plugins/codecompletion/codecompletion.cpp (revision 5973)
+++ src/plugins/codecompletion/codecompletion.cpp (working copy)
@@ -172,7 +172,8 @@
     m_IsAutoPopup(false),
     m_ToolbarChanged(true),
     m_CurrentLine(0),
-    m_LastFile(wxEmptyString)
+    m_LastFile(wxEmptyString),
+    m_NeedReparse(false)
 {
     if(!Manager::LoadResource(_T("codecompletion.zip")))
     {
@@ -2069,7 +2070,11 @@
             }
         }
     }
-
+    if ((event.GetModificationType() & wxSCI_MOD_INSERTTEXT) ||
+        (event.GetModificationType() & wxSCI_MOD_DELETETEXT))
+    {
+        m_NeedReparse = true;
+    }
     if( control->GetCurrentLine() != m_CurrentLine)
     {
         m_CurrentLine = control->GetCurrentLine();
@@ -2093,6 +2098,16 @@
                 m_Scope->SetSelection(wxNOT_FOUND);
             }
         }
+        if (m_NeedReparse)
+        {
+            Parser* parser = m_NativeParser.FindParserFromActiveEditor();
+            if (parser)
+            {
+                parser->Reparse(editor->GetFilename());
+            }
+            m_NeedReparse= false;
+        }
+
     }
 
     // allow others to handle this event
Index: src/plugins/codecompletion/codecompletion.h
===================================================================
--- src/plugins/codecompletion/codecompletion.h (revision 5973)
+++ src/plugins/codecompletion/codecompletion.h (working copy)
@@ -146,7 +146,7 @@
         int                                m_CurrentLine;
         map<wxString, int>                 m_SearchItem;
         wxString                           m_LastFile;
-
+        bool                               m_NeedReparse;
         bool                               m_LexerKeywordsToInclude[9];
 
         DECLARE_EVENT_TABLE()
Index: src/plugins/codecompletion/nativeparser.cpp
===================================================================
--- src/plugins/codecompletion/nativeparser.cpp (revision 5973)
+++ src/plugins/codecompletion/nativeparser.cpp (working copy)
@@ -2152,18 +2152,13 @@
     }
     s_LastEditor = editor;
     s_LastLine = line;
-
-    Parser parser(this);
-    parser.ParseBufferForFunctions(control->GetTextRange(0, pos));
-
-    wxArrayString funcs;
-    TokensTree* tmptree = parser.GetTempTokens();
-
-    // look for implementation functions that enclose our current line number
-    for(size_t i = 0; i < tmptree->size();i++)
+    TokenIdxSet result;
+    m_Parser.FindTokensInFile(editor->GetFilename(), result, tkFunction|tkConstructor|tkDestructor);
+    TokensTree* tree = m_Parser.GetTokens();
+    for (TokenIdxSet::iterator it = result.begin(); it != result.end(); ++it)
     {
-        Token* token = tmptree->at(i);
-        if (token && (token->m_TokenKind == tkFunction || token->m_TokenKind == tkConstructor || token->m_TokenKind == tkDestructor))
+        Token* token = tree->at(*it);
+        if (token)
         {
             // found a function; check its bounds
             if (token->m_ImplLineStart <= (size_t)line && token->m_ImplLineEnd >= (size_t)line)
@@ -2174,6 +2169,7 @@
                                                                 token->DisplayName().wx_str(),
                                                                 token->m_ImplLine));
 
+
                 s_LastNS = token->GetNamespace();
                 s_LastPROC = token->m_Name;
                 s_LastResult = control->PositionFromLine(token->m_ImplLine - 1);
Index: src/plugins/codecompletion/parser/parser.cpp
===================================================================
--- src/plugins/codecompletion/parser/parser.cpp (revision 5973)
+++ src/plugins/codecompletion/parser/parser.cpp (working copy)
@@ -538,7 +538,7 @@
             }
 
             if (!opts.loader) //this should always be true (memory will leak if a loader has already been initialized before this point)
-                opts.loader = Manager::Get()->GetFileManager()->Load(bufferOrFilename, false);
+                opts.loader = Manager::Get()->GetFileManager()->Load(bufferOrFilename, true);
         }
 
 #if PARSER_DEBUG_OUTPUT
@@ -1045,3 +1045,22 @@
     }
     return true;
 }
+size_t Parser::FindTokensInFile(const wxString& fileName,TokenIdxSet& result,short int kindMask)
+{
+    result.clear();
+    wxString file = UnixFilename(fileName);
+    TokenIdxSet tmpresult;
+    wxCriticalSectionLocker lock(s_MutexProtection);
+    if(!m_pTokens->FindTokensInFile(file,tmpresult,kindMask))
+        return 0;
+
+    TokenIdxSet::iterator it;
+    for(it = tmpresult.begin();it!=tmpresult.end();++it)
+    {
+        Token* token = m_pTokens->at(*it);
+        if(token)
+        //result.push_back(token);
+        result.insert(*it);
+    }
+    return result.size();
+}
Index: src/plugins/codecompletion/parser/parser.h
===================================================================
--- src/plugins/codecompletion/parser/parser.h (revision 5973)
+++ src/plugins/codecompletion/parser/parser.h (working copy)
@@ -153,6 +153,7 @@
         Token* FindChildTokenByName(Token* parent, const wxString& name, bool useInheritance = false, short int kindMask = 0xFFFF) const;
         size_t FindMatches(const wxString& s, TokenList&   result, bool caseSensitive = true, bool is_prefix = true);
         size_t FindMatches(const wxString& s, TokenIdxSet& result, bool caseSensitive = true, bool is_prefix = true);
+        size_t FindTokensInFile(const wxString& fileName, TokenIdxSet& result, short int kindMask);
         ParserOptions& Options() { return m_Options; }
         BrowserOptions& ClassBrowserOptions() { return m_BrowserOptions; }
 
Index: src/sdk/filemanager.cpp
===================================================================
--- src/sdk/filemanager.cpp (revision 5973)
+++ src/sdk/filemanager.cpp (working copy)
@@ -138,12 +138,8 @@
                 cbEditor* ed = em->GetBuiltinEditor(em->GetEditor(i));
                 if(ed && fileName == ed->GetFilename())
                 {
-                    wxString s(ed->GetControl()->GetText());
-                    #if wxCHECK_VERSION(2, 9, 0)
-                    NullLoader *nl = new NullLoader(file, (char*) s.wx_str(), s.length() * sizeof(wxChar));
-                    #else
-                    NullLoader *nl = new NullLoader(file, (char*) s.c_str(), s.length() * sizeof(wxChar));
-                    #endif
+
+                    EditorReuser *nl = new EditorReuser(file, ed->GetControl()->GetText());
                     return nl;
                 }
             }


hi,morten :
you are right.so the patch above should be changed.
« Last Edit: January 07, 2010, 12:38:12 pm by blueshake »
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: make cc real-time parse
« Reply #16 on: January 07, 2010, 12:57:22 pm »
optimize the cc search process based on real-time parse.
[...]
Code
Index: src/include/filemanager.h
===================================================================
--- src/plugins/codecompletion/nativeparser.cpp (revision 5973)
+++ src/plugins/codecompletion/nativeparser.cpp (working copy)
@@ -2152,18 +2152,13 @@
     }
     s_LastEditor = editor;
     s_LastLine = line;
-
-    Parser parser(this);
-    parser.ParseBufferForFunctions(control->GetTextRange(0, pos));
-
-    wxArrayString funcs;
-    TokensTree* tmptree = parser.GetTempTokens();
-
-    // look for implementation functions that enclose our current line number
-    for(size_t i = 0; i < tmptree->size();i++)
+    TokenIdxSet result;
+    m_Parser.FindTokensInFile(editor->GetFilename(), result, tkFunction|tkConstructor|tkDestructor);
+    TokensTree* tree = m_Parser.GetTokens();
+    for (TokenIdxSet::iterator it = result.begin(); it != result.end(); ++it)
     {
-        Token* token = tmptree->at(i);
-        if (token && (token->m_TokenKind == tkFunction || token->m_TokenKind == tkConstructor || token->m_TokenKind == tkDestructor))
+        Token* token = tree->at(*it);
+        if (token)
         {
             // found a function; check its bounds
             if (token->m_ImplLineStart <= (size_t)line && token->m_ImplLineEnd >= (size_t)line)
@@ -2174,6 +2169,7 @@
                                                                 token->DisplayName().wx_str(),
                                                                 token->m_ImplLine));
 
+
                 s_LastNS = token->GetNamespace();
                 s_LastPROC = token->m_Name;
                 s_LastResult = control->PositionFromLine(token->m_ImplLine - 1);
This is certainly the cause for the wrong file being returned which I've reported here:
http://forums.codeblocks.org/index.php/topic,11800.msg80076.html#msg80076
Look: Previously the parser was created based on the native parsers current state. Now you just use the attached parser which maybe wrong because it may point to the wrong file!
I believe something like FindParserFromEditor needs to be used here...

I check out the FindParserFromEditor.It just simply return the m_Parser.
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: make cc real-time parse
« Reply #17 on: January 07, 2010, 01:13:44 pm »
I check out the FindParserFromEditor.It just simply return the m_Parser.
You are right. What a messy code that was... From the early stages where there were several parsers per project. I've cleaned it up thoroughly and committed.

Ps.: While I was at it I fixed the bug with the wrong file pointer... ;-)
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ