Tokenizer should return a token type?

[ollydbg]

Here, I found several months ago, I have write this suggestion Re: New code completion remarks/issues

Today, I found more evidence, see here:

http://www.macs.hw.ac.uk/~alison/alg/lectures/l7.pdf

Here is the test in this pdf:

The tokeniser should extract both the text
(lexeme) and the class of the item. So, a token
for “35” should contain the text (35) and the
type NUM.

also

So.. a suitable datatype for a token will be a
struct or class such as the following:
struct token {
char* text;
tokentype type;
};
where tokentype is an enumerated type
specifying possible types of token.

So, a token type is necessary.


By the way, this site:
The Mini C++ Interpreter give a simple example of a C++ parser. There is a chapter describing this mini C++ interpreter in the book "The art of C++".