Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => CodeCompletion redesign => Topic started by: ollydbg on February 22, 2010, 02:57:10 pm

Title: ParserTester for codecompletion plugin
Post by: ollydbg on February 22, 2010, 02:57:10 pm
we(Loaden, blueshake and visualfc and ollydbg) were pleased to announce a parser tester project, this project contains some source under codecompletion/parser folder, the log message can be shown in the simple Gui. This is really convenient when you debug parser related code, since debugging the CC within codeblocks Gui is quite slow(gdb need to load the host exe file and many other dlls, it's too slow)

Here are the steps to use this project.

1, You just extract the files to the parser folder(I only change the tokenizer and parserthread source to support the ParserTester project, adding some preprocessor directive), then open the parsertest.cbp.

2, build

3, run, then, all the "TRACE" message will be shown in the ParserTester mainframe.

By default, you need to supply a "test.cpp" in the same folder for the tester to parse

A log.txt will be saved after testing.







[attachment deleted by admin]
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 22, 2010, 02:59:15 pm
Also, I have move the "handling conditional preprocessor code to Tokenizer class".

Here, I think it is better to move the conditional preprocessor related code from parserthread to tokenizer class.

At the moment, this was done in the parserthread, so we need some specific code to check whether there is a #if, this is even worse when like

Code
void function( A a, B b
#ifdef XXXX
, C c)
#else
, C c, D d)
#endif
,  then,
Code
( A a, B b #ifdef XXXX, C c)
will be returned as a whole token.because In the Tokenizer, a pair of braces will be returned as a whole token, thus made conditional directive unbalanced.

for example :CC fails to parse parts of boost (http://forums.codeblocks.org/index.php/topic,11909.0.html)

for another example, in the source of parserThread.cpp, handling class, we need to check the #if statement like:

Code
 // handle preprocessor directives in class definition, e.g.

class MyClass
        #ifdef FOO
             : public MyClass1
             , public MyClass2
         #endif
{}

This modification will let the Tokenizer give a transparent and consistent way to feed the parserthread.
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on February 22, 2010, 03:53:10 pm
 :D
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 22, 2010, 03:53:31 pm
Another improvement of the parser, I have fix two bugs in parsing VC 2008/2005 header files. (Both the STL and the window API functions can auto prompted correctly).

bug one: In the Visual C++ 's header for std::string, there are some statement:

Code
    enum
        {    // length of internal buffer, [1, 16]
        _BUF_SIZE = 16 / sizeof (_Elem) < 1 ? 1
            : 16 / sizeof(_Elem)};

Note the "<", in parser will try the skip the block of "<", but badly, there's no ">" here, so, the parser just skip to the EOF.

bug two:

In the windows API heade file, there's code statement like below:
Code
__inline
BOOL
GetMessage(
    LPMSG lpMsg,
    HWND hWnd,
    UINT wMsgFilterMin,
    UINT wMsgFilterMax
    )
{
#ifdef UNICODE
    return GetMessageW(
#else
    return GetMessageA(
#endif
        lpMsg,
        hWnd,
        wMsgFilterMin,
        wMsgFilterMax
        );
}
the parser failed because there are #if in the function arguments.
All the two bugs were reporteded by Loaden.
Title: Re: ParserTester for codecompletion plugin
Post by: MortenMacFly on February 22, 2010, 05:06:10 pm
Well I don't really get what's the state of this work... you got me confused. :(

Does this include the patches by blueshake concerning #define handling or not? And what about the other modification concerning pointer / reference?

If you want to do me a favour, please bring in new pieces step-by-step and mark what's new.
Title: Re: ParserTester for codecompletion plugin
Post by: JGM on February 22, 2010, 07:07:47 pm
Wow great initiative by you guys, this will definitely improve CC development and improvements by you, the most brave guys on codeblock forums  :)
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 23, 2010, 01:39:40 am
@morten:
Sorry for my poor English explanation. I don't have patch of blueshake included in my package.
@JGM:
Thanks for the encouragement.

The aim of this parser tester project was that all the "TRACE" logs were redirected to a standalone simple App instead of the "Debug Log" panel in the C::B host program. So, when you want to check whether the parsing(which involve the parserthread and tokenizer class) works correctly, you can check these log messages.

parsertest.cbp contains several source directly from the CC's source, in fact, I just add two extra source files. one is "parsertest.cpp" which create a simple GUI window, the other is "parser2.cpp" which is a minimal mimic of the "parser.cpp". All the header files were from CC source.

1, You download the ParsertesterV1.zip package, and extract the files to the "src/plugin/codecompletion/parser" folder, then open the parsertest.cbp.

2, build this project, also, you need to supply a file named "test.cpp" as the source file to be parsed.

3, run the generated APP, You will see, all the "TRACE" message were shown in the ParserTester mainframe.

4, A log.txt will be saved after testing, you can exam it.

Note:
To let the parsertest.cbp build successfully, I have change the Macros in the front of parserthread.cpp and tokenizer.cpp. For example:
In the parserthread.cpp

Code
#ifdef PARSER_TEST
  extern void ParserTrace(const wxChar* format, ...);
  #define TRACE(format, args...)\
  ParserTrace(format , ## args)
#else
#if PARSERTHREAD_DEBUG_OUTPUT
   #define TRACE(format, args...)\
   Manager::Get()->GetLogManager()->DebugLog(F( format , ## args))
#else
   #define TRACE(format, args...)
#endif
#endif

So, the TRACE was replaces by the "ParserTrace" routing, which do the hack. This modification doesn't interrupt building the normal Codecompletion plugin, so the inference to the CC's source is minimal.

Also, this package, I have other improvements in the Tokenizer class.( even you don't have these improvement, you can still compile and build the parsertest.cbp)

one improvement is that I add "handling conditional preprocessor" in the Tokenizer::SkipUnwanted function. so, The first thing we are checking is if the CurrentChar is "#", we need to handle the "#if #else #elif #endif". I comment out these code in Tokenizer::DoGetToken()
Code
//    else if (c == '(')
//    {
//        m_IsOperator = false;
//
//        // skip blocks () []
//        if (!SkipBlock(CurrentChar()))
//            return wxEmptyString;
//
//        str = FixArgument(m_Buffer.Mid(start, m_TokenIndex - start));
//        CompactSpaces(str);
//    }

Because these code snippet never consider the #if like statement in the function argument.

Instead, I add another function called : Tokenizer::DoAdvanceGetToken
Code
wxString Tokenizer::DoAdvanceGetToken()
{
    wxString str = DoGetToken();
    if ( str == _T("(") )
    {
        int braceLevel = 1;

        wxString temp;
        TokenizerState undoTokenizerState;
        undoTokenizerState = m_State;
        m_State = tsSkipNone;
        do
        {
            temp = DoGetToken();
            str << temp;
            if ( temp == _T("(") )
            {
                braceLevel++;
            }
            else if (temp == _T(")"))
            {
                braceLevel--;
            }
        }
        while( braceLevel>0  && (!temp.IsEmpty()) );
        m_State = undoTokenizerState;
    }
    return str;
}
This is because the "(" and ")" should always be matched, so we still return a "(XXXX,YYYY)", but the conditional preprocessor code inside the function were stripped.

Here is another improvement, these improvement is only for parsing VC 2008/2005's header file correctly, as I stated in my previous posts.


Edit
We have only tested this "parsertest" project under Windows.
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on February 23, 2010, 07:11:37 am
My English is not good, the following screenshots to compare this with the patch with the SVN version of the difference.
Compiler: VC10RC
Figure 1-5 is the SVN test, Figure 6-10 is applied after this patch.
I hope that parsertest be able to merge into the CC, so that more people easy to maintain CC.
I wish CC stronger!
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on February 23, 2010, 07:12:33 am
5-8:
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on February 23, 2010, 07:15:09 am
9、10
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 23, 2010, 07:20:44 am
wow, this is a visual comparison of my patch! Thanks Loaden.
Title: Re: ParserTester for codecompletion plugin
Post by: MortenMacFly on February 23, 2010, 08:05:48 am
9、10
I don't get what you want to tell with the pictures. Can you phrase what this means? Please keep in mind that spamming pictures in our forum wastes a lot of forum-space. In addition they will sooner or later be deleted. At that time your posts are meaningless.

Please: Write what you have to tell.
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on February 23, 2010, 08:10:41 am
Okay. What I mean is: Use this patch, you can perfect solution to VC's STL code completion, as well as the API Question. In contrast pictures will find a lot of tokens increases, which would make the CC header file for VC support even better!
Well, pictures can be deleted.
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on February 23, 2010, 08:21:02 am
Important! To test the VC compiler header files also need to replace the two macro rules:
Code
				<_STD_BEGIN>
<![CDATA[-namespace std {]]>
</_STD_BEGIN>
<_STD_END>
<![CDATA[}]]>
</_STD_END>
Title: Re: ParserTester for codecompletion plugin
Post by: MortenMacFly on February 23, 2010, 09:14:39 am
Important! To test the VC compiler header files also need to replace the two macro rules:
...replace with what? Do you mean remove?
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 23, 2010, 09:56:59 am
@morten:
Loaden means to parse the VC header files correctly, We need to add two rules in the wxstring-wxstring replacement map.

Title: Re: ParserTester for codecompletion plugin
Post by: blueshake on February 23, 2010, 12:18:18 pm
@Loaden

if you want to show pictures,you can use outside repository.then cite the picture addresses here. :D
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 25, 2010, 06:15:29 am
In the ParserTest.cpp, if you would like to add the replacement tokens, use these statement:

Code
void Start()
    {
        Parser client(NULL);
        wxString fileName = _T("test.cpp");
        FileLoader* loader = new FileLoader(fileName);
        (*loader)();

        TokensTree* tree = new TokensTree();

        Tokenizer::SetReplacementString(_T("_GLIBCXX_STD"),                    _T("std"));

        Tokenizer::SetReplacementString(_T("_GLIBCXX_BEGIN_NESTED_NAMESPACE"), _T("+namespace"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_END_NESTED_NAMESPACE"),   _T("}"));

        Tokenizer::SetReplacementString(_T("_GLIBCXX_BEGIN_NAMESPACE"),        _T("+namespace"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_END_NAMESPACE"),          _T("}"));

        Tokenizer::SetReplacementString(_T("_GLIBCXX_END_NAMESPACE_TR1"),      _T("}"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_BEGIN_NAMESPACE_TR1"),    _T("-namespace tr1 {"));

        // for VC2005/2008
        Tokenizer::SetReplacementString(_T("_STD_BEGIN"),    _T("-namespace std {"));
        Tokenizer::SetReplacementString(_T("_STD_END"),      _T("}"));


        ParserThreadOptions opts;
        opts.wantPreprocessor = false;
        opts.useBuffer = false;
        opts.bufferSkipBlocks = false;
        opts.bufferSkipOuterBlocks = false;
        opts.followLocalIncludes = false;
        opts.followGlobalIncludes = false;
        opts.loader = loader;

        ParserThread* ph = new ParserThread(&client, fileName, true, opts, tree);
        bool b = ph->Parse();
        delete ph;

        ShowLog();
    }
Title: Re: ParserTester for codecompletion plugin
Post by: MortenMacFly on February 25, 2010, 07:28:25 am
In the ParserTest.cpp, if you would like to add the replacement tokens, use these statement:
Why not the others, too? (The ones from the actual plugin.)

Edit: Argh, forget about it: I missed the scrollbar in your post. :lol:
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 25, 2010, 07:45:02 am
Edit: Argh, forget about it: I missed the scrollbar in your post. :lol:
:D, these two replacement rules should be added to the standard CC plug-in too.

By the way, Loaden suggests that we could have different replacement string set for different compiler. :D
Title: Re: ParserTester for codecompletion plugin
Post by: MortenMacFly on February 26, 2010, 06:07:49 pm
By the way, Loaden suggests that we could have different replacement string set for different compiler. :D
I wouldn't do that by the compiler. The reasons are:

1.) CC needs to "talk" to the compiler which may not be present. CC should also work without the compiler (the compiler is another plugin in the end).
2.) You can use e.g. Visual C++ SDK's with GCC just fine (like Direct/X for example). So the replacements do in fact not depend on the compiler as you see.

Better proposal: Try to "identify" commonly used SDK's (like base SDK's) upon typical identifiers in the header files For example: If you encounter _GLIBCXX_STD switch to the right set of replacements tokens and re-parse. If you encounter _STD_BEGIN switch (or better append) this set.

Notice that you can also use GCC / VC libraries / header files mixed. So a combination os replacements tokens may make sense.
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 27, 2010, 04:37:04 am
Notice that you can also use GCC / VC libraries / header files mixed. So a combination os replacements tokens may make sense.
Ok, in this situation, both replacement string for GCC and VC should be used.
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 27, 2010, 04:50:16 am
Here is a improved parserTester version 2.
A lot of GUI improved by Loaden. Added a "find" non-model dialog.
Also, we have moved the project sources to a separate folder instead of the "/plugins/codecompletion/parser"

So, you need to download the 7z package, and unzip it. it should have a "ptest" folder, you need to paste this folder (include all the files) to "codecompletion/parser", so, now, all the source files were in "codecompletion/parser/ptest".

Also, a you need to apply these patches to let the "parserTester" project compile successfully. ( This patch never hurt the building of standard CC)

In this patch, I have fix this problem: Insert all class method without implementation question (http://forums.codeblocks.org/index.php/topic,12091.0.html)

Comments are welcome!!!

By the way, I don't change these code snippet in tokenizer.cpp ,but the TortoiseSVN always thought I was changing that...
Code
#ifdef __WXMSW__ // This is a Windows only bug!
    else if (c == 178 || c == 179 || c == 185) // fetch ² and ³
    {
        str = c;
        MoveToNextChar();
    }
#endif


I'm not sure why.






[attachment deleted by admin]
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on February 27, 2010, 06:43:22 am
Sorry, my fault, in "ParserTesterV2.7z" compressed package, the "mainfdsfef.cpp" is unnecessary, and could be deleted.
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on February 27, 2010, 11:45:43 am
Refactoring code, and add two methods: PrintTree and PrintList.
It should work better.

[attachment deleted by admin]
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on February 27, 2010, 01:31:38 pm
Refactoring code, and add two methods: PrintTree and PrintList.
It should work better.
Great! It works better now! Thanks for your work!
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on March 02, 2010, 03:43:11 pm
Hi,all, I created a unix project for ptest, and after testing, it works well.
Another change: I changed test.cpp TO test.h, so that it can be added to the project, for easy testing.

[attachment deleted by admin]
Title: Re: ParserTester for codecompletion plugin
Post by: code robot on March 13, 2010, 09:55:19 am
THANK YOU!!!

I can now code again using the VC++ compiler :)

Please add this to the main repo a.s.a.p.
(In case anyone else needs this fix right away, I manually copy/pasted the .cpp/.h files from 'parserTesterV1.zip' , into my local SVN checkout folder, and recompiled Code::Blocks)
Title: Re: ParserTester for codecompletion plugin
Post by: Loaden on March 21, 2010, 06:27:08 am
By the way, I don't change these code snippet in tokenizer.cpp ,but the TortoiseSVN always thought I was changing that...
Code
#ifdef __WXMSW__ // This is a Windows only bug!
    else if (c == 178 || c == 179 || c == 185) // fetch ² and ³
    {
        str = c;
        MoveToNextChar();
    }
#endif


I find the reason: don't use TortoiseSVN , you can use patch in MSYS.
Title: Re: ParserTester for codecompletion plugin
Post by: ollydbg on May 12, 2010, 04:24:50 pm
This is a pure patch to let the parserthread.cpp and tokenizer.cpp supporting ParserTester project.

Code
Index: parserthread.cpp
===================================================================
--- parserthread.cpp (revision 6218)
+++ parserthread.cpp (working copy)
@@ -21,12 +21,18 @@
 
 #define PARSERTHREAD_DEBUG_OUTPUT 0
 
+#ifdef PARSER_TEST
+  extern void ParserTrace(const wxChar* format, ...);
+  #define TRACE(format, args...)\
+  ParserTrace(format , ## args)
+#else
 #if PARSERTHREAD_DEBUG_OUTPUT
-    #define TRACE(format, args...)\
-    Manager::Get()->GetLogManager()->DebugLog(F( format , ## args))
+   #define TRACE(format, args...)\
+   Manager::Get()->GetLogManager()->DebugLog(F( format , ## args))
 #else
-    #define TRACE(format, args...)
+   #define TRACE(format, args...)
 #endif
+#endif
 
 int THREAD_START       = wxNewId();
 int THREAD_END         = wxNewId();
@@ -1184,11 +1190,13 @@
         while (!token.IsEmpty() && token != ParserConsts::kw_endif)
             token = m_Tokenizer.GetToken();
         --m_PreprocessorIfCount;
-#if PARSERTHREAD_DEBUG_OUTPUT
+#if PARSERTHREAD_DEBUG_OUTPUT || defined PARSER_TEST
         int l = m_Tokenizer.GetNestingLevel();
 #endif
         m_Tokenizer.RestoreNestingLevel();
+#if PARSERTHREAD_DEBUG_OUTPUT || defined PARSER_TEST
         TRACE(_T("HandlePreprocessorBlocks() : Restoring nesting level: %d (was %d)"), m_Tokenizer.GetNestingLevel(), l);
+#endif
     }
     else if (preproc==ParserConsts::kw_endif) // #endif
         --m_PreprocessorIfCount;
Index: tokenizer.cpp
===================================================================
--- tokenizer.cpp (revision 6218)
+++ tokenizer.cpp (working copy)
@@ -15,15 +15,22 @@
 #include "manager.h"
 #include <cctype>
 #include <globals.h>
+#include "logmanager.h"
 
 #define TOKENIZER_DEBUG_OUTPUT 0
 
+#ifdef PARSER_TEST
+    extern void ParserTrace(const wxChar* format, ...);
+    #define TRACE(format, args...)\
+    ParserTrace(format , ## args)
+#else
 #if TOKENIZER_DEBUG_OUTPUT
     #define TRACE(format, args...)\
     Manager::Get()->GetLogManager()->DebugLog(F( format , ## args))
 #else
     #define TRACE(format, args...)
 #endif
+#endif
 
 namespace TokenizerConsts
 {