ParserTester for codecompletion plugin

[ollydbg]

@morten:
Loaden means to parse the VC header files correctly, We need to add two rules in the wxstring-wxstring replacement map.

[blueshake]

@Loaden

if you want to show pictures,you can use outside repository.then cite the picture addresses here.

[ollydbg]

In the ParserTest.cpp, if you would like to add the replacement tokens, use these statement:

Code

void Start()
    {
        Parser client(NULL);
        wxString fileName = _T("test.cpp");
        FileLoader* loader = new FileLoader(fileName);
        (*loader)();

        TokensTree* tree = new TokensTree();

        Tokenizer::SetReplacementString(_T("_GLIBCXX_STD"),                    _T("std"));

        Tokenizer::SetReplacementString(_T("_GLIBCXX_BEGIN_NESTED_NAMESPACE"), _T("+namespace"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_END_NESTED_NAMESPACE"),   _T("}"));

        Tokenizer::SetReplacementString(_T("_GLIBCXX_BEGIN_NAMESPACE"),        _T("+namespace"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_END_NAMESPACE"),          _T("}"));

        Tokenizer::SetReplacementString(_T("_GLIBCXX_END_NAMESPACE_TR1"),      _T("}"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_BEGIN_NAMESPACE_TR1"),    _T("-namespace tr1 {"));

        // for VC2005/2008
        Tokenizer::SetReplacementString(_T("_STD_BEGIN"),    _T("-namespace std {"));
        Tokenizer::SetReplacementString(_T("_STD_END"),      _T("}"));


        ParserThreadOptions opts;
        opts.wantPreprocessor = false;
        opts.useBuffer = false;
        opts.bufferSkipBlocks = false;
        opts.bufferSkipOuterBlocks = false;
        opts.followLocalIncludes = false;
        opts.followGlobalIncludes = false;
        opts.loader = loader;

        ParserThread* ph = new ParserThread(&client, fileName, true, opts, tree);
        bool b = ph->Parse();
        delete ph;

        ShowLog();
    }

[MortenMacFly]

In the ParserTest.cpp, if you would like to add the replacement tokens, use these statement:

Why not the others, too? (The ones from the actual plugin.)

Edit: Argh, forget about it: I missed the scrollbar in your post. :lol:

[ollydbg]

Edit: Argh, forget about it: I missed the scrollbar in your post. :lol:

, these two replacement rules should be added to the standard CC plug-in too.

By the way, Loaden suggests that we could have different replacement string set for different compiler.

[MortenMacFly]

By the way, Loaden suggests that we could have different replacement string set for different compiler.

I wouldn't do that by the compiler. The reasons are:

1.) CC needs to "talk" to the compiler which may not be present. CC should also work without the compiler (the compiler is another plugin in the end).
2.) You can use e.g. Visual C++ SDK's with GCC just fine (like Direct/X for example). So the replacements do in fact not depend on the compiler as you see.

Better proposal: Try to "identify" commonly used SDK's (like base SDK's) upon typical identifiers in the header files For example: If you encounter _GLIBCXX_STD switch to the right set of replacements tokens and re-parse. If you encounter _STD_BEGIN switch (or better append) this set.

Notice that you can also use GCC / VC libraries / header files mixed. So a combination os replacements tokens may make sense.

[ollydbg]

Notice that you can also use GCC / VC libraries / header files mixed. So a combination os replacements tokens may make sense.

Ok, in this situation, both replacement string for GCC and VC should be used.

[ollydbg]

Here is a improved parserTester version 2.
A lot of GUI improved by Loaden. Added a "find" non-model dialog.
Also, we have moved the project sources to a separate folder instead of the "/plugins/codecompletion/parser"

So, you need to download the 7z package, and unzip it. it should have a "ptest" folder, you need to paste this folder (include all the files) to "codecompletion/parser", so, now, all the source files were in "codecompletion/parser/ptest".

Also, a you need to apply these patches to let the "parserTester" project compile successfully. ( This patch never hurt the building of standard CC)

In this patch, I have fix this problem: Insert all class method without implementation question

Comments are welcome!!!

By the way, I don't change these code snippet in tokenizer.cpp ,but the TortoiseSVN always thought I was changing that...

Code

#ifdef __WXMSW__ // This is a Windows only bug!
    else if (c == 178 || c == 179 || c == 185) // fetch ² and ³
    {
        str = c;
        MoveToNextChar();
    }
#endif

I'm not sure why.






[attachment deleted by admin]

[Loaden]

Sorry, my fault, in "ParserTesterV2.7z" compressed package, the "mainfdsfef.cpp" is unnecessary, and could be deleted.

[Loaden]

Refactoring code, and add two methods: PrintTree and PrintList.
It should work better.

[attachment deleted by admin]

[ollydbg]

Refactoring code, and add two methods: PrintTree and PrintList.
It should work better.

Great! It works better now! Thanks for your work!

[Loaden]

Hi,all, I created a unix project for ptest, and after testing, it works well.
Another change: I changed test.cpp TO test.h, so that it can be added to the project, for easy testing.

[attachment deleted by admin]

[code robot]

THANK YOU!!!

I can now code again using the VC++ compiler

Please add this to the main repo a.s.a.p.
(In case anyone else needs this fix right away, I manually copy/pasted the .cpp/.h files from 'parserTesterV1.zip' , into my local SVN checkout folder, and recompiled Code::Blocks)

[Loaden]

By the way, I don't change these code snippet in tokenizer.cpp ,but the TortoiseSVN always thought I was changing that...

Code

#ifdef __WXMSW__ // This is a Windows only bug!
    else if (c == 178 || c == 179 || c == 185) // fetch ² and ³
    {
        str = c;
        MoveToNextChar();
    }
#endif

I find the reason: don't use TortoiseSVN , you can use patch in MSYS.

[ollydbg]

This is a pure patch to let the parserthread.cpp and tokenizer.cpp supporting ParserTester project.

Code

Index: parserthread.cpp
===================================================================
--- parserthread.cpp	(revision 6218)
+++ parserthread.cpp	(working copy)
@@ -21,12 +21,18 @@
 
 #define PARSERTHREAD_DEBUG_OUTPUT 0
 
+#ifdef PARSER_TEST
+  extern void ParserTrace(const wxChar* format, ...);
+  #define TRACE(format, args...)\
+  ParserTrace(format , ## args)
+#else
 #if PARSERTHREAD_DEBUG_OUTPUT
-    #define TRACE(format, args...)\
-    Manager::Get()->GetLogManager()->DebugLog(F( format , ## args))
+   #define TRACE(format, args...)\
+   Manager::Get()->GetLogManager()->DebugLog(F( format , ## args))
 #else
-    #define TRACE(format, args...)
+   #define TRACE(format, args...)
 #endif
+#endif
 
 int THREAD_START       = wxNewId();
 int THREAD_END         = wxNewId();
@@ -1184,11 +1190,13 @@
         while (!token.IsEmpty() && token != ParserConsts::kw_endif)
             token = m_Tokenizer.GetToken();
         --m_PreprocessorIfCount;
-#if PARSERTHREAD_DEBUG_OUTPUT
+#if PARSERTHREAD_DEBUG_OUTPUT || defined PARSER_TEST
         int l = m_Tokenizer.GetNestingLevel();
 #endif
         m_Tokenizer.RestoreNestingLevel();
+#if PARSERTHREAD_DEBUG_OUTPUT || defined PARSER_TEST
         TRACE(_T("HandlePreprocessorBlocks() : Restoring nesting level: %d (was %d)"), m_Tokenizer.GetNestingLevel(), l);
+#endif
     }
     else if (preproc==ParserConsts::kw_endif) // #endif
         --m_PreprocessorIfCount;
Index: tokenizer.cpp
===================================================================
--- tokenizer.cpp	(revision 6218)
+++ tokenizer.cpp	(working copy)
@@ -15,15 +15,22 @@
 #include "manager.h"
 #include <cctype>
 #include <globals.h>
+#include "logmanager.h"
 
 #define TOKENIZER_DEBUG_OUTPUT 0
 
+#ifdef PARSER_TEST
+    extern void ParserTrace(const wxChar* format, ...);
+    #define TRACE(format, args...)\
+    ParserTrace(format , ## args)
+#else
 #if TOKENIZER_DEBUG_OUTPUT
     #define TRACE(format, args...)\
     Manager::Get()->GetLogManager()->DebugLog(F( format , ## args))
 #else
     #define TRACE(format, args...)
 #endif
+#endif
 
 namespace TokenizerConsts
 {