Intergrate the nativeparser_base to our parsertest project

[ollydbg]

When review our CC code, I see that the core type matching code is:

Code

    // find all other matches
    std::queue<ParserComponent> components;
    BreakUpComponents(actual_search, components);

    m_LastAISearchWasGlobal = components.size() <= 1;
    if (!components.empty())
        m_LastAIGlobalSearch = components.front().component;

    ResolveExpression(tree, components, *search_scope, result, caseSensitive, isPrefix);

    if (s_DebugSmartSense)
        CCLogger::Get()->DebugLog(F(_T("AI() AI leave, returned %d results"),result.size()));

    return result.size();

The components is the statement when user write.
The ResolveExpression is in Nativeparser_base now. So, I think we can safely add the nativeparser_base to our parsertest project.

I see that the class Nativeparser_base is quite nice for testing. It does not contains UI related code. (simply remove #include <cbstyledtextctrl.h> in the cpp file)

I'm going to write the test for the ResolveExpression.(mostly after the parsing stage)

E.g.

Code

class A
{
public:
int m_Member1;
int m_Member2;
};

class B:public A
{
};

B obj;

Now, we have a expression "obj.m_Member1".
The initial search scope is the -1 (the global namespace)

Running the function: ResolveExpression should return the one exact matches, which is the Token "m_Member1".
So, this is only the start point, later we can test more complex expressions or more classes in the hierarchy or template related code.

I'm starting the work on my local git repos. I will report any progress here in this thread.

[ollydbg]

Ok, it works nice.

Copy nativeparser_base.h and nativeparser_base.cpp to the parser folder.

I have add some code snippet in frame.cpp

Code

    //Here we are going to test the expression solving algorithm

    NativeParserTest nativeParserTest;

    wxString exp = _T("obj.m_Member1");

    TokenIdxSet searchScope;
    searchScope.insert(-1);

    TokenIdxSet result;

    TokensTree *tree = ParserTest::Get()->GetTokensTree();

    nativeParserTest.TestExpression(exp,
                                    tree,
                                    searchScope,
                                    result );

    wxLogMessage(_T("Result have %d matches"), result.size());


    for (TokenIdxSet::iterator it=result.begin(); it!=result.end(); ++it)
    {
        Token* token = tree->at(*it);
        if (token)
        {
            wxString log;
            log << token->GetTokenKindString() << _T(" ")
                << token->DisplayName()        << _T("\t[")
                << token->m_Line               << _T(",")
                << token->m_ImplLine           << _T("]");
            CCLogger::Get()->Log(log);
        }
    }

And here is the class declaration and implementation:

Code

#ifndef NATIVEPARSERTEST_H
#define NATIVEPARSERTEST_H

#include "nativeparser_base.h"

class NativeParserTest : public NativeParserBase
{
public:
    NativeParserTest();
    ~NativeParserTest();
    bool TestExpression(wxString& expression,
                        TokensTree * tree,
                        const TokenIdxSet& searchScope,
                        TokenIdxSet&   result);
};

#endif //NATIVEPARSERTEST_H

Code

#include <sdk.h>

#ifndef CB_PRECOMP
#endif

#include "nativeparsertest.h"

#include "parser/cclogger.h"



#define CC_NATIVEPARSERTEST_DEBUG_OUTPUT 0

#if CC_GLOBAL_DEBUG_OUTPUT == 1
    #undef CC_NATIVEPARSERTEST_DEBUG_OUTPUT
    #define CC_NATIVEPARSERTEST_DEBUG_OUTPUT 1
#elif CC_GLOBAL_DEBUG_OUTPUT == 2
    #undef CC_NATIVEPARSERTEST_DEBUG_OUTPUT
    #define CC_NATIVEPARSERTEST_DEBUG_OUTPUT 2
#endif

#ifdef CC_PARSER_TEST
//    #define ADDTOKEN(format, args...) \
//            CCLogger::Get()->AddToken(F(format, ##args))
//    #define TRACE(format, args...) \
//            CCLogger::Get()->DebugLog(F(format, ##args))
//    #define TRACE2(format, args...) \
//            CCLogger::Get()->DebugLog(F(format, ##args))

    #define ADDTOKEN(format, args...) \
            wxLogMessage(F(format, ##args))
    #define TRACE(format, args...) \
            wxLogMessage(F(format, ##args))
    #define TRACE2(format, args...) \
            wxLogMessage(F(format, ##args))
#else
    #if CC_NATIVEPARSERTEST_DEBUG_OUTPUT == 1
        #define ADDTOKEN(format, args...) \
                CCLogger::Get()->AddToken(F(format, ##args))
        #define TRACE(format, args...) \
            CCLogger::Get()->DebugLog(F(format, ##args))
        #define TRACE2(format, args...)
    #elif CC_NATIVEPARSERTEST_DEBUG_OUTPUT == 2
        #define ADDTOKEN(format, args...) \
                CCLogger::Get()->AddToken(F(format, ##args))
        #define TRACE(format, args...)                                              \
            do                                                                      \
            {                                                                       \
                if (g_EnableDebugTrace)                                             \
                    CCLogger::Get()->DebugLog(F(format, ##args));                   \
            }                                                                       \
            while (false)
        #define TRACE2(format, args...) \
            CCLogger::Get()->DebugLog(F(format, ##args))
    #else
        #define ADDTOKEN(format, args...)
        #define TRACE(format, args...)
        #define TRACE2(format, args...)
    #endif
#endif


extern bool s_DebugSmartSense;

NativeParserTest::NativeParserTest( )
{

}
NativeParserTest::~NativeParserTest()
{

}
bool NativeParserTest::TestExpression(wxString& expression,
                                      TokensTree * tree,
                                      const TokenIdxSet& searchScope,
                                      TokenIdxSet&   result)
{
    // find all other matches
    std::queue<ParserComponent> components;
    BreakUpComponents(expression, components);

    ResolveExpression(tree, components, searchScope, result, true, false);

    if (s_DebugSmartSense)
        CCLogger::Get()->DebugLog(F(_T("NativeParserTest::TestExpression , returned %d results"),result.size()));

    return true;
}

The test code is in my previous post, and I can see the final result:

...
000033. NativeParserTest::TestExpression , returned 1 results
000034. variable int A::m_Member1   [5,0]
...

EDIT:
I add the diff file, so it is test only patch.(Note, the diff file contains my other part of changes to CC, like adding some comments, re-direct the TRACE message to standard console, so I can still see these messages when the app hit a breakpoint.....)
The patch is generated by git, but I think using the patch command under msys, it can directly apply the change on a SVN trunk.
BTW: Maybe, some EOL should be changed after patching, because my git use Linux-style EOL.

[MortenMacFly]

The ResolveExpression is in Nativeparser_base now. So, I think we can safely add the nativeparser_base to our parsertest project.
[...]
I see that the class Nativeparser_base is quite nice for testing. It does not contains UI related code.

That was the overall goal of this re-factoring. ;-)

Ok, it works nice.
[...]
I add the diff file, so it is test only patch.

Good to know - I didn't have the time to test, but I believe there is even more we can transfer to the test project, if we include a cbStyledTextCtrl into it.

[MortenMacFly]

The patch is generated by git, but I think using the patch command under msys, it can directly apply the change on a SVN trunk.

Nope, doesn't work here. So I cannot try. Maybe you can do a proper svn diff-based patch. Its easy: just transfer the changes to the svn controlled folder and do a svn diff > patch in the root folder of Code::Blocks's sources.

[Jenna]

I also use git for my local repo and creating svn compatible  patches needs just one more step: see http://abombss.com/blog/2007/12/10/creating-patches-for-subversion-from-git/, or use the attached python script (if it works on windows).
It's not created by me, I found it somewhere in the internet (do not recall where).

[MortenMacFly]

I also use git for my local repo and creating svn compatible  patches needs just one more step: [...] or use the attached python script (if it works on windows).

Right... I'll try that but this becomes really annoying! I mean: Whats the point in pre-processing patches in a platform dependent way, if we can have proper patches done right using subversion?

[Jenna]

I also use git for my local repo and creating svn compatible  patches needs just one more step: [...] or use the attached python script (if it works on windows).

Right... I'll try that but this becomes really annoying! I mean: Whats the point in pre-processing patches in a platform dependent way, if we can have proper patches done right using subversion?

I use git for my local workspace, because it is more flexible, if I work with several patches, because I can easily create local branches.
And running one script after creating a patch to be applied on windows is not too much work for me.
On linux the git-created patches can be easily applied using the "patch" executable without any further processing.

[oBFusCATed]

At work I use git-svn-diff script, it generates proper windows svn patches, but as far as I know it doesn't work on windows...

[MortenMacFly]

I use git for my local workspace, because it is more flexible,

Surely one can use locally whatever (s)he thinks is right. My point is: I would like to see that even if people use GIT or whatever locally that we post patches here created by svn diff. This will cause no hassle on all platforms - it just works. Everything else is a nightmare... at least on Windows.

Besides: Even after installing (updating) Python and running the script, applying the modified patch still doesn't work and errors with my favourite error message "chunk info expected". So I give up on this.

[ollydbg]

The patch is generated by git, but I think using the patch command under msys, it can directly apply the change on a SVN trunk.

Nope, doesn't work here. So I cannot try. Maybe you can do a proper svn diff-based patch. Its easy: just transfer the changes to the svn controlled folder and do a svn diff > patch in the root folder of Code::Blocks's sources.

Here it is.

It was created from the plugins folder.

BTW: I can still use the command :
patch -p1 < aaa.patch
to apply the patch (generated in git )on svs local copy.

[oBFusCATed]

ollydbg:
What patch executable are you using?
All I've tried crashed badly when used with linux generated patches.

[ollydbg]

ollydbg:
What patch executable are you using?
All I've tried crashed badly when used with linux generated patches.

I use the patch command shipped with MSYS.
I always download the MSYS zip file from mingw64 site(some one has maintain a full msys environment which contains everything), see:MSYS-20111123.zip

[MortenMacFly]

Here it is.

Thanks, this works. BTW: I still need to modify it if you don't do the diff from the root of trunk. When you are using a subversion 1.7 working copy, applying a patch only works well from root and then the path's are not correct. However, I could handle this...

Just looking at the diff I realised you copied the nativeparser_base into the parser folder. I wouldn't do this. Already now they are out of sync. I think a better solution is to either move nativeparser_base into the "parser" sub-folder or move the project file of parsertest one level up.

Your changes conflict a little with the ones of my working copy, but I can handle. For the record, I have touched in parserthread:
- GetTokenBaseType
- HandleFunction
So once you are touching those, we should sync.

[ollydbg]

Here it is.

Thanks, this works. BTW: I still need to modify it if you don't do the diff from the root of trunk. When you are using a subversion 1.7 working copy, applying a patch only works well from root and then the path's are not correct. However, I could handle this...

Sorry about this, but handling different patches is a problem for me(that's why I start using the git), because I have some other changes to the cb_trunk in my local copy, so it is a mess up when I need to select all the changed files in TortoiseSVN, and generate a diff file. For me, the simplest way is just select a folder like "codecompletion", and generate the diff file.

Just looking at the diff I realised you copied the nativeparser_base into the parser folder. I wouldn't do this. Already now they are out of sync. I think a better solution is to either move nativeparser_base into the "parser" sub-folder or move the project file of parsertest one level up.

I suggest move the nativeparer_base to the sub-folder, it is all related to c/c++ parser.

Your changes conflict a little with the ones of my working copy, but I can handle. For the record, I have touched in parserthread:
- GetTokenBaseType
- HandleFunction
So once you are touching those, we should sync.

All my changes were record in my git local clone, so I can safely pull the changes you made, and merge it.

[MortenMacFly]

so it is a mess up when I need to select all the changed files in TortoiseSVN, and generate a diff file. For me, the simplest way is just select a folder like "codecompletion", and generate the diff file.

If you are using TortoiseSVN IMHO is should be possible even from trunk to only select the files in question. If not, I am using SmartSVN where you have a "flat view", meaning that the files are shown in a big list as if there were not directories from the root folder. Then you can select only the files you want to create a diff from. I am not sure if TortoiseSVN has a similar feature.

I suggest move the nativeparer_base to the sub-folder, it is all related to c/c++ parser.

Fine with me.

All my changes were record in my git local clone, so I can safely pull the changes you made, and merge it.

The are not "ready for production" yet.