Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => CodeCompletion redesign => Topic started by: ollydbg on April 13, 2010, 04:40:37 pm

Title: Tokenizer should return a token type?
Post by: ollydbg on April 13, 2010, 04:40:37 pm

Here, I found several months ago, I have write this suggestion Re: New code completion remarks/issues (http://forums.codeblocks.org/index.php/topic,11187.msg79616.html#msg79616)

Today, I found more evidence, see here:

http://www.macs.hw.ac.uk/~alison/alg/lectures/l7.pdf

Here is the test in this pdf:

Quote

The tokeniser should extract both the text
(lexeme) and the class of the item. So, a token
for “35” should contain the text (35) and the
type NUM.

also

Quote

So.. a suitable datatype for a token will be a
struct or class such as the following:
struct token {
char* text;
tokentype type;
};
where tokentype is an enumerated type
specifying possible types of token.

So, a token type is necessary. :D


By the way, this site:
The Mini C++ Interpreter (http://www.excode.ru/art5843.html) give a simple example of a C++ parser. There is a chapter describing this mini C++ interpreter in the book "The art of C++".