Developer forums (C::B DEVELOPMENT STRICTLY!) > CodeCompletion redesign

Tokenizer should return a token type?

(1/1)

ollydbg:
Here, I found several months ago, I have write this suggestion Re: New code completion remarks/issues

Today, I found more evidence, see here:

http://www.macs.hw.ac.uk/~alison/alg/lectures/l7.pdf

Here is the test in this pdf:

--- Quote ---The tokeniser should extract both the text
(lexeme) and the class of the item. So, a token
for “35” should contain the text (35) and the
type NUM.
--- End quote ---

also

--- Quote ---So.. a suitable datatype for a token will be a
struct or class such as the following:
struct token {
char* text;
tokentype type;
};
where tokentype is an enumerated type
specifying possible types of token.
--- End quote ---

So, a token type is necessary. :D


By the way, this site:
The Mini C++ Interpreter give a simple example of a C++ parser. There is a chapter describing this mini C++ interpreter in the book "The art of C++".

Navigation

[0] Message Index