Here, I found several months ago, I have write this suggestion
Re: New code completion remarks/issuesToday, I found more evidence, see here:
http://www.macs.hw.ac.uk/~alison/alg/lectures/l7.pdfHere is the test in this pdf:
The tokeniser should extract both the text
(lexeme) and the class of the item. So, a token
for “35” should contain the text (35) and the
type NUM.
also
So.. a suitable datatype for a token will be a
struct or class such as the following:
struct token {
char* text;
tokentype type;
};
where tokentype is an enumerated type
specifying possible types of token.
So, a token type is necessary.
By the way, this site:
The Mini C++ Interpreter give a simple example of a C++ parser. There is a chapter describing this mini C++ interpreter in the book "The art of C++".