User forums > Help

Codecompletion parser bug on treating comments

<< < (3/3)

ollydbg:
@Ceniza
I do agree with you on the idea of a "smart lexer".

The current implementation of the lexer was located in Tokenizer.h and Tokenizer.cpp, they just return a simple wxString, and no tag information was attached. So, the parser(Syntax Analyzer, I also use this term in the wiki Code Completion design) will do the whole work(most of the syntax analysis was done in parseThread.cpp and parseThread.h)

As a "smart lexer", it will feed parser not only a simple wxString, but an object indication if they are strings, numbers, identifiers. Oh, I nearly forget, the "smart lexer" will also do the preprocessing stage, which is also very important :D

ollydbg:

--- Quote from: MortenMacFly on March 10, 2009, 07:08:38 am ---Something else that came into my mind just by now:
Why don't we kind of "pre-process" the buffer before CC analyses it in term of removing comments completely. I mean: Commented stuff is just useless for CC (unless we want to consider using Doxygen comments or alike) and probably operating the whole buffer could work with a "simple" RegEx?! In the end we would obsolete a lot of comment checking code.

--- End quote ---

Comment checking code is not quite difficult.
I checked the source code. Each time we call Tokenzier::DoGetToken(), it will first try to strip the comment


--- Code: ---wxString Tokenizer::DoGetToken()
{
    if (IsEOF())
        return wxEmptyString;

    if (!SkipWhiteSpace())
        return wxEmptyString;

    if (m_SkipUnwantedTokens && !SkipUnwanted())       // ****************Here
        return wxEmptyString;

    // if m_SkipUnwantedTokens is false, we need to handle comments here too
    if (!m_SkipUnwantedTokens)
        SkipComment();                                 //*****************Here
    ........

--- End code ---

If m_SkipUnwantedTokens is true (which is a normal situation), then the SkipComment() will be called in the SkipUnwanted() function.

If m_SkipUnwantedTokens is false (which means we are in a special situation, such as eating the argument of a templates, we shouldn't call SkipUnwanted(), then SkipComment() will be called manually).

With all the steps before, I think Comment can be stripped quite well. :D Am I right? If wrong, please correct me. Thank you!

Navigation

[0] Message Index

[*] Previous page