Author Topic: ParserTester for codecompletion plugin  (Read 17268 times)

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5219
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
ParserTester for codecompletion plugin
« on: February 22, 2010, 02:57:10 pm »
we(Loaden, blueshake and visualfc and ollydbg) were pleased to announce a parser tester project, this project contains some source under codecompletion/parser folder, the log message can be shown in the simple Gui. This is really convenient when you debug parser related code, since debugging the CC within codeblocks Gui is quite slow(gdb need to load the host exe file and many other dlls, it's too slow)

Here are the steps to use this project.

1, You just extract the files to the parser folder(I only change the tokenizer and parserthread source to support the ParserTester project, adding some preprocessor directive), then open the parsertest.cbp.

2, build

3, run, then, all the "TRACE" message will be shown in the ParserTester mainframe.

By default, you need to supply a "test.cpp" in the same folder for the tester to parse

A log.txt will be saved after testing.







[attachment deleted by admin]
« Last Edit: February 22, 2010, 03:50:35 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5219
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: ParserTester for codecompletion plugin
« Reply #1 on: February 22, 2010, 02:59:15 pm »
Also, I have move the "handling conditional preprocessor code to Tokenizer class".

Here, I think it is better to move the conditional preprocessor related code from parserthread to tokenizer class.

At the moment, this was done in the parserthread, so we need some specific code to check whether there is a #if, this is even worse when like

Code: [Select]
void function( A a, B b
#ifdef XXXX
, C c)
#else
, C c, D d)
#endif
,  then,
Code: [Select]
( A a, B b #ifdef XXXX, C c) will be returned as a whole token.because In the Tokenizer, a pair of braces will be returned as a whole token, thus made conditional directive unbalanced.

for example :CC fails to parse parts of boost

for another example, in the source of parserThread.cpp, handling class, we need to check the #if statement like:

Code: [Select]
// handle preprocessor directives in class definition, e.g.

class MyClass
        #ifdef FOO
             : public MyClass1
             , public MyClass2
         #endif
{}

This modification will let the Tokenizer give a transparent and consistent way to feed the parserthread.
« Last Edit: February 22, 2010, 03:06:52 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: ParserTester for codecompletion plugin
« Reply #2 on: February 22, 2010, 03:53:10 pm »
 :D

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5219
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: ParserTester for codecompletion plugin
« Reply #3 on: February 22, 2010, 03:53:31 pm »
Another improvement of the parser, I have fix two bugs in parsing VC 2008/2005 header files. (Both the STL and the window API functions can auto prompted correctly).

bug one: In the Visual C++ 's header for std::string, there are some statement:

Code: [Select]
    enum
        {    // length of internal buffer, [1, 16]
        _BUF_SIZE = 16 / sizeof (_Elem) < 1 ? 1
            : 16 / sizeof(_Elem)};

Note the "<", in parser will try the skip the block of "<", but badly, there's no ">" here, so, the parser just skip to the EOF.

bug two:

In the windows API heade file, there's code statement like below:
Code: [Select]
__inline
BOOL
GetMessage(
    LPMSG lpMsg,
    HWND hWnd,
    UINT wMsgFilterMin,
    UINT wMsgFilterMax
    )
{
#ifdef UNICODE
    return GetMessageW(
#else
    return GetMessageA(
#endif
        lpMsg,
        hWnd,
        wMsgFilterMin,
        wMsgFilterMax
        );
}
the parser failed because there are #if in the function arguments.
All the two bugs were reporteded by Loaden.
« Last Edit: February 22, 2010, 04:00:39 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9493
Re: ParserTester for codecompletion plugin
« Reply #4 on: February 22, 2010, 05:06:10 pm »
Well I don't really get what's the state of this work... you got me confused. :(

Does this include the patches by blueshake concerning #define handling or not? And what about the other modification concerning pointer / reference?

If you want to do me a favour, please bring in new pieces step-by-step and mark what's new.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline JGM

  • Lives here!
  • ****
  • Posts: 518
  • Got to practice :)
Re: ParserTester for codecompletion plugin
« Reply #5 on: February 22, 2010, 07:07:47 pm »
Wow great initiative by you guys, this will definitely improve CC development and improvements by you, the most brave guys on codeblock forums  :)

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5219
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: ParserTester for codecompletion plugin
« Reply #6 on: February 23, 2010, 01:39:40 am »
@morten:
Sorry for my poor English explanation. I don't have patch of blueshake included in my package.
@JGM:
Thanks for the encouragement.

The aim of this parser tester project was that all the "TRACE" logs were redirected to a standalone simple App instead of the "Debug Log" panel in the C::B host program. So, when you want to check whether the parsing(which involve the parserthread and tokenizer class) works correctly, you can check these log messages.

parsertest.cbp contains several source directly from the CC's source, in fact, I just add two extra source files. one is "parsertest.cpp" which create a simple GUI window, the other is "parser2.cpp" which is a minimal mimic of the "parser.cpp". All the header files were from CC source.

1, You download the ParsertesterV1.zip package, and extract the files to the "src/plugin/codecompletion/parser" folder, then open the parsertest.cbp.

2, build this project, also, you need to supply a file named "test.cpp" as the source file to be parsed.

3, run the generated APP, You will see, all the "TRACE" message were shown in the ParserTester mainframe.

4, A log.txt will be saved after testing, you can exam it.

Note:
To let the parsertest.cbp build successfully, I have change the Macros in the front of parserthread.cpp and tokenizer.cpp. For example:
In the parserthread.cpp

Code: [Select]
#ifdef PARSER_TEST
  extern void ParserTrace(const wxChar* format, ...);
  #define TRACE(format, args...)\
  ParserTrace(format , ## args)
#else
#if PARSERTHREAD_DEBUG_OUTPUT
   #define TRACE(format, args...)\
   Manager::Get()->GetLogManager()->DebugLog(F( format , ## args))
#else
   #define TRACE(format, args...)
#endif
#endif

So, the TRACE was replaces by the "ParserTrace" routing, which do the hack. This modification doesn't interrupt building the normal Codecompletion plugin, so the inference to the CC's source is minimal.

Also, this package, I have other improvements in the Tokenizer class.( even you don't have these improvement, you can still compile and build the parsertest.cbp)

one improvement is that I add "handling conditional preprocessor" in the Tokenizer::SkipUnwanted function. so, The first thing we are checking is if the CurrentChar is "#", we need to handle the "#if #else #elif #endif". I comment out these code in Tokenizer::DoGetToken()
Code: [Select]
//    else if (c == '(')
//    {
//        m_IsOperator = false;
//
//        // skip blocks () []
//        if (!SkipBlock(CurrentChar()))
//            return wxEmptyString;
//
//        str = FixArgument(m_Buffer.Mid(start, m_TokenIndex - start));
//        CompactSpaces(str);
//    }

Because these code snippet never consider the #if like statement in the function argument.

Instead, I add another function called : Tokenizer::DoAdvanceGetToken
Code: [Select]
wxString Tokenizer::DoAdvanceGetToken()
{
    wxString str = DoGetToken();
    if ( str == _T("(") )
    {
        int braceLevel = 1;

        wxString temp;
        TokenizerState undoTokenizerState;
        undoTokenizerState = m_State;
        m_State = tsSkipNone;
        do
        {
            temp = DoGetToken();
            str << temp;
            if ( temp == _T("(") )
            {
                braceLevel++;
            }
            else if (temp == _T(")"))
            {
                braceLevel--;
            }
        }
        while( braceLevel>0  && (!temp.IsEmpty()) );
        m_State = undoTokenizerState;
    }
    return str;
}
This is because the "(" and ")" should always be matched, so we still return a "(XXXX,YYYY)", but the conditional preprocessor code inside the function were stripped.

Here is another improvement, these improvement is only for parsing VC 2008/2005's header file correctly, as I stated in my previous posts.


Edit
We have only tested this "parsertest" project under Windows.
« Last Edit: February 23, 2010, 01:45:04 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: ParserTester for codecompletion plugin
« Reply #7 on: February 23, 2010, 07:11:37 am »
My English is not good, the following screenshots to compare this with the patch with the SVN version of the difference.
Compiler: VC10RC
Figure 1-5 is the SVN test, Figure 6-10 is applied after this patch.
I hope that parsertest be able to merge into the CC, so that more people easy to maintain CC.
I wish CC stronger!
« Last Edit: March 19, 2010, 06:04:00 am by Loaden »

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: ParserTester for codecompletion plugin
« Reply #8 on: February 23, 2010, 07:12:33 am »
5-8:
« Last Edit: March 19, 2010, 06:04:17 am by Loaden »

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: ParserTester for codecompletion plugin
« Reply #9 on: February 23, 2010, 07:15:09 am »
9、10
« Last Edit: March 19, 2010, 06:04:31 am by Loaden »

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5219
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: ParserTester for codecompletion plugin
« Reply #10 on: February 23, 2010, 07:20:44 am »
wow, this is a visual comparison of my patch! Thanks Loaden.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9493
Re: ParserTester for codecompletion plugin
« Reply #11 on: February 23, 2010, 08:05:48 am »
9、10
I don't get what you want to tell with the pictures. Can you phrase what this means? Please keep in mind that spamming pictures in our forum wastes a lot of forum-space. In addition they will sooner or later be deleted. At that time your posts are meaningless.

Please: Write what you have to tell.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: ParserTester for codecompletion plugin
« Reply #12 on: February 23, 2010, 08:10:41 am »
Okay. What I mean is: Use this patch, you can perfect solution to VC's STL code completion, as well as the API Question. In contrast pictures will find a lot of tokens increases, which would make the CC header file for VC support even better!
Well, pictures can be deleted.

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: ParserTester for codecompletion plugin
« Reply #13 on: February 23, 2010, 08:21:02 am »
Important! To test the VC compiler header files also need to replace the two macro rules:
Code: [Select]
<_STD_BEGIN>
<![CDATA[-namespace std {]]>
</_STD_BEGIN>
<_STD_END>
<![CDATA[}]]>
</_STD_END>

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9493
Re: ParserTester for codecompletion plugin
« Reply #14 on: February 23, 2010, 09:14:39 am »
Important! To test the VC compiler header files also need to replace the two macro rules:
...replace with what? Do you mean remove?
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ