A patch adding a state variable to Tokenizer class

[ollydbg]

Welcome to test it.

1, I add a state variable to Tokenizer, and this will make the code more clearer. So, we can still use the GetToken() and PeekToken() with out any argument. also, I just re-factor the SkipUnWanted() function.
2, add a attempt to deal variable seperated by commas problem.(these related code is developed by blueshake).



[attachment deleted by admin]

[MortenMacFly]

Mmmmh... a few things are not clear to me:

For example: Why did you remove theis code:

Code

#ifdef __WXMSW__ // This is a Windows only bug!
    else if (c == 178 || c == 179 || c == 185) // fetch ² and ³
    {
        str = c;
        MoveToNextChar();
    }
#endif

This re-introduces a critical crash-bug on Windows?!

You've removed quite some portions... could you explain a little more why exactly? Or did you have a difference code base?

[ollydbg]

@morten:
Sorry, That's my fault. That's becaus In my Chinese Windows XP, I can't see the charactor well, sometimes, there will be shown a rectangle or a question mark. So, that's was a little annoying. So I delete that code. :wink:

Is there other question or comments?

[MortenMacFly]

Is there other question or comments?

Sure. ;-)

The other thing is: Your re-factored the SkipUnwanted(bool bGetValue) quite heavily including the removal of handling of preprocessor directives (which makes m_IsPreprocessor and m_LastPreprocessor obsolete btw.). Why? This was working great?!

Finally shouldn't statements like:

Code

if ( m_State & tsSkipQuestion )

better be:

Code

if ( (m_State & tsSkipQuestion) == tsSkipQuestion)

(Need to have a look myself to see how the bits are defined though...)

[ollydbg]

The other thing is: Your re-factored the SkipUnwanted(bool bGetValue) quite heavily including the removal of handling of preprocessor directives (which makes m_IsPreprocessor and m_LastPreprocessor obsolete btw.). Why? This was working great?!

Handle preprocessor is quite easy in ParserThread, So, I remove the handling preprocessor related code in Tokenizer, I don't think they are needed any more. Maybe, we need more test.