Author Topic: A patch adding a state variable to Tokenizer class  (Read 11715 times)

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
A patch adding a state variable to Tokenizer class
« on: December 29, 2009, 07:48:55 am »
Welcome to test it.

1, I add a state variable to Tokenizer, and this will make the code more clearer. So, we can still use the GetToken() and PeekToken() with out any argument. also, I just re-factor the SkipUnWanted() function.
2, add a attempt to deal variable seperated by commas problem.(these related code is developed by blueshake).



[attachment deleted by admin]
« Last Edit: December 29, 2009, 03:12:40 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9693
Re: A patch adding a state variable to Tokenizer class
« Reply #1 on: December 31, 2009, 02:41:34 pm »
Mmmmh... a few things are not clear to me:

For example: Why did you remove theis code:
Code
#ifdef __WXMSW__ // This is a Windows only bug!
    else if (c == 178 || c == 179 || c == 185) // fetch ² and ³
    {
        str = c;
        MoveToNextChar();
    }
#endif
???

This re-introduces a critical crash-bug on Windows?!

You've removed quite some portions... could you explain a little more why exactly? Or did you have a difference code base?
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: A patch adding a state variable to Tokenizer class
« Reply #2 on: January 01, 2010, 03:28:34 pm »
@morten:
Sorry, That's my fault. That's becaus In my Chinese Windows XP, I can't see the charactor well, sometimes, there will be shown a rectangle or a question mark. So, that's was a little annoying. So I delete that code. :wink:

Is there other question or comments?
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9693
Re: A patch adding a state variable to Tokenizer class
« Reply #3 on: January 01, 2010, 07:44:05 pm »
Is there other question or comments?
Sure. ;-)

The other thing is: Your re-factored the SkipUnwanted(bool bGetValue) quite heavily including the removal of handling of preprocessor directives (which makes m_IsPreprocessor and m_LastPreprocessor obsolete btw.). Why? This was working great?!

Finally shouldn't statements like:
Code
if ( m_State & tsSkipQuestion )
better be:
Code
if ( (m_State & tsSkipQuestion) == tsSkipQuestion)
(Need to have a look myself to see how the bits are defined though...)
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: A patch adding a state variable to Tokenizer class
« Reply #4 on: January 02, 2010, 02:48:18 am »
The other thing is: Your re-factored the SkipUnwanted(bool bGetValue) quite heavily including the removal of handling of preprocessor directives (which makes m_IsPreprocessor and m_LastPreprocessor obsolete btw.). Why? This was working great?!

Handle preprocessor is quite easy in ParserThread, So, I remove the handling preprocessor related code in Tokenizer, I don't think they are needed any more. :D Maybe, we need more test.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.