Author Topic: Tokenizer should return a token type?  (Read 6137 times)

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5721
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Tokenizer should return a token type?
« on: April 13, 2010, 04:40:37 pm »
Here, I found several months ago, I have write this suggestion Re: New code completion remarks/issues

Today, I found more evidence, see here:

Here is the test in this pdf:
The tokeniser should extract both the text
(lexeme) and the class of the item. So, a token
for “35” should contain the text (35) and the
type NUM.

So.. a suitable datatype for a token will be a
struct or class such as the following:
struct token {
char* text;
tokentype type;
where tokentype is an enumerated type
specifying possible types of token.

So, a token type is necessary. :D

By the way, this site:
The Mini C++ Interpreter give a simple example of a C++ parser. There is a chapter describing this mini C++ interpreter in the book "The art of C++".
« Last Edit: April 13, 2010, 04:46:11 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.