Code completion doesnt follow #include in struct

[zacaj]

I posted this in the bug tracker, and was told to post here

object.h:

Code

typedef struct
{
#include "object_struct.h"
} Object;

object_struct.h:

Code

    vec3f pos; ///< The 3D position of the object
    vec3f lpos; ///< The 3D position of the object last update
    float r; ///< The radius of the Object
    uchar type; ///< Used to identify if the Object has been inherited
	Quaternion q; ///< The rotation of the Object, as a Quaternion
	void *data; ///< Pointer to the data used by this object for rendering

main.c:

Code

#include "object.h"

int main()
{
  Object *object=malloc(sizeof(Object));
  object->ty//HERE
}

after typing ty at //HERE type will not be suggested, nor will any other member of Object

This is even more annoying if you have another object 'inherit' Object:
enemy.h:

Code

typedef struct
{
  #include "object_struct.h"
  float hp;
} Enemy

main.c:

Code

#include "enemy.h"

int main()
{
  Enemy *enemy=malloc(sizeof(Enemy));
  object->//HERE
}

Code completion will automatically complete with hp (I know this can be turned off, but its useful in lots of other cases)

[oBFusCATed]

C::B version, OS version?
If C::B is <=10.05 then try some of the latest nightlies...

[ollydbg]

@zacaj
Ok, I know why cc does not show it's member.
Because Currently, CC's parser dose not do a full preprocessor. I mean, both files were parsed separately, so, the members in "object_struct.h" will be added to "global namespace" instead of the struct. 

It is too hard to implement a full parser. so the bug can't finished soon. The most reasonal way in the feature was: gcc/clang, those two were all compilers, so they do a full parse on your code.

[zacaj]

C::B rev 6992, Windows 7 32 bit

Is the CC parser custom?  A full preprocessor does seem like it would be complex, but it doesnt seem (at least in my mind, I havent looked at the code) like it should be that hard to jump into another file if its included and just keep reading from there

[oBFusCATed]

zacaj: Why don't you use proper inheritance? Yes, C has it, too.

It is something like:

Code

typedef struct A
{
     members of A;
};

typedef struct B {
    A base;
    members of B;
};

[zacaj]

Because that looks really weird.  I use it in less used classes, but for something as common as Object, I dont want to have to type an extra ->base-> every time I change the position

[ollydbg]

Because that looks really weird.  I use it in less used classes, but for something as common as Object, I dont want to have to type an extra ->base-> every time I change the position

I think we can not solve your problem unless we use a full preprocessor/parser framework. 

[JGM]

some weeks ago I started working on a simple to use cpp parser mainly a prepreocessor just for fun. I worked on it about 2 days and was fixing some special cases where the parser wouldn't parse some things as it should do, so some code is commented out since I was debugging it, I was developing it on ubuntu and only using c/c++ standard libraries. I was programming it on a way that the preprocessor would return the final code as it should look for normal parsing and each elements on the code tokenized and identified by enums. Also I did some function place holders for expressions parsing and other things left to do. So if someone is interested on checking it out I uploaded the code to mediafire on a zip file:

http://www.mediafire.com/?yqvsstq23jot650

Comments and suggestions are welcome, I stopped after reading about clang with all it's advanced features, but if someone thinks this could be useful a spark of motivation may come to light xD (i gave up but then thought that the code may be of some use, and I think I was implementing it on a way not so hard to maintain for the future, but after all I'm a noob xD)

Edit: whoa I reached post 500 

[ollydbg]

nice work, I am downloading your code and do some checking.

I know clang was a full preprocessor/parser framework. but it was too complex, and it was released under BSD style license. And finally I switch my mind to gcc. I have read the gcc's cpp internal manual (development manual), and found that developing a preprocessor was really complex.

the advantage of a full preprocessor/parser framework is that it has one AST tree for every translation unit, so, it is precise, but slow.
and currently cc's implementation is: parse every file only once, and collect every token to a single tree, just like ctags did, even there are some files missing, cc's parser still do some guess and continue parsing.

for a preprocessor framework, I remembered that One c::b developer  has implemented a preprocessor(but it was hard to read for me)

[ollydbg]

Comments and suggestions are welcome, I stopped after reading about clang with all it's advanced features, but if someone thinks this could be useful a spark of motivation may come to light xD (i gave up but then thought that the code may be of some use, and I think I was implementing it on a way not so hard to maintain for the future, but after all I'm a noob xD)

here are my two point
1, I think using a generated lexer will make things much easier. a generated lexer can handle somethings like: line counting, column counting, and it use a state machine which will catch the "keyword" much faster than "clang or gcc". (both clang and gcc does not distinguish between a keyword or an identifier, they just do a hashtable search when an identifier returned), from this point, I'd suggest my work on Quex based lexer. ( I do benchmarks showing that it was 200% as the speed of flex generated lexer under windows). I put the test code here( also it include a clang test project to test codecompletion feature of clang )
http://code.google.com/p/quexparser/
would you like to have a look?

2, I found you use std::vector in the code, does std::list is much better? when doing a macro replacement, a vector will always re-size itself.

Currently, I feel a little confused about my quexparser, I do not have a clean direction, I found that even doing a macro replacement need many tricky.
you can look at
http://gcc.gnu.org/onlinedocs/cppinternals/

[oBFusCATed]

2, I found you use std::vector in the code, does std::list is much better? when doing a macro replacement, a vector will always re-size itself.

Or probably a std::deque -> http://www.gotw.ca/publications/mill10.htm

[MortenMacFly]

some weeks ago I started working on a simple to use cpp parser mainly a prepreocessor just for fun.

[...]

http://www.mediafire.com/?yqvsstq23jot650

I cannot test it atm, but do you mean "a preprocessor", or "for preprocessors?

My recent idea concerning preprocessor was using tools like:
http://dotat.at/prog/unifdef/
(...and there are other going in the same direction like "sunifdef") to "clean up" source files in a pre-process and parse what's left. If we do this in memory it should also be pretty fast.

[JGM]

...
1, I think using a generated lexer will make things much easier. a generated lexer can handle somethings like: line counting, column counting, and it use a state machine which will catch the "keyword" much faster than "clang or gcc". (both clang and gcc does not distinguish between a keyword or an identifier, they just do a hashtable search when an identifier returned), from this point, I'd suggest my work on Quex based lexer. ( I do benchmarks showing that it was 200% as the speed of flex generated lexer under windows). I put the test code here( also it include a clang test project to test codecompletion feature of clang )
http://code.google.com/p/quexparser/
would you like to have a look?
...

I started writing a custom tokenizer for the preprocessor since I thought the output would be much simple to analyze on the future and also I wanted it to be smart and produce a tree more easy to analyze. Also I wanted to produce the cleaned code after preproccessing with correct column and line numbers for the code parser as optimizable if possible. Still I need to correctly manage multiple line preprocessors (#define blah blah(123) \).

Whoa! that quexparser looks a little kind of complex for my brain to digest I will try to analyze it deeply.

...
2, I found you use std::vector in the code, does std::list is much better? when doing a macro replacement, a vector will always re-size itself.
...

2. I have read several c++ books and read about the performance on available containers as inner structure but I always forget the differences on each of them (lack of practice) but it may be easy to substitute since containers almost always share same interface (I think)

Currently, I feel a little confused about my quexparser, I do not have a clean direction, I found that even doing a macro replacement need many tricky.
you can look at
http://gcc.gnu.org/onlinedocs/cppinternals/

Yep, this whole c++ parsing thing is hard since the language itself has so many features to look up, but it is fun, I just wanted to create a simple to use preprocessor after several months of c++ inactivity on my blood.

[JGM]

some weeks ago I started working on a simple to use cpp parser mainly a prepreocessor just for fun.

[...]

http://www.mediafire.com/?yqvsstq23jot650

I cannot test it atm, but do you mean "a preprocessor", or "for preprocessors?

My recent idea concerning preprocessor was using tools like:
http://dotat.at/prog/unifdef/
(...and there are other going in the same direction like "sunifdef") to "clean up" source files in a pre-process and parse what's left. If we do this in memory it should also be pretty fast.

yep a preprocessor, with the future goal of complete parser. (my english vocabulary and game of words suck  )

Current logic is to identify preprocessor type when tokenizing (function ex: #define test(x) (x*2) or just a declaration ex #define test_delcared) add it to a vector to then make correct replacements on code to parse it correctly. Actually nested preprocessors as I tested worked correctly, I was fixing some issues with multiple line preprocessors (handle incorrectly to produce correct line and column positions) and then write an expression parser to evaluate macro expressions. Also include files are parsed only once as normally, it handles global and local includes and you can indicate to the class the paths to search. Also the use of string class should be replaced by the wstring one, but since I was playing around at first there are things to be improved.

I wanted to have a much complete and documented code before posting it but well, after reading some threads here I decided to let it go as it is and see if it is understandable by other developers, wishing for the best. The main.cpp file should serve as an example of how I intended to make use of it. If people think the code is not that hard to understand then it may merit it's completeness.

[ollydbg]

some weeks ago I started working on a simple to use cpp parser mainly a prepreocessor just for fun.

[...]

http://www.mediafire.com/?yqvsstq23jot650

I cannot test it atm, but do you mean "a preprocessor", or "for preprocessors?

My recent idea concerning preprocessor was using tools like:
http://dotat.at/prog/unifdef/
(...and there are other going in the same direction like "sunifdef") to "clean up" source files in a pre-process and parse what's left. If we do this in memory it should also be pretty fast.

I briefly read the site: unifdef - selectively remove C preprocessor conditionals
it said:

It is useful for avoiding distractions when studying code that uses #ifdef heavily for portability (the original motivation was xterm's pty handling code), or as a lightweight preprocessor to strip out internal routines from a public header (the Linux kernel uses unifdef to strip out #ifdef __KERNEL__ sections from the headers it exports to userland)

Great, I think we need a lightweight preprocessor, as my point of view, gcc's preprocessor code base was too big and too complex.
The main two job is:
1, handle conditional preprocessor directive, like #if  and do a expression evaluation.
2, do macro expansion

In fact this two method was done in the current implementation of cc, but I think they need to be refactored. Morten, can you give a direction?

@JGM
quex's lexer generator is quite easy to lean, and it's grammar is very easy to learn. once you use this, you can give(retern) a token once a time. the token contains several information include at least four field.
1, token id (identifier, keyword, open-bracket......)
2, string value if it is an identifier, otherwise, it is empty
3, column count value
4, line count value

then you don't care about anything else, you just use the token, and do everything you like. So, quex's lexer stands on a low level, and you can implement the high level preprocessor on that.

I have implement a const value expression solver by "shunting yard algorithm" on the code. There is a quite similar one in the CC's source code. We can have further discussion to collaborate.

[JGM]

So basically you created a definition file http://code.google.com/p/quexparser/source/browse/trunk/cppparser/cpp.qx and the quex library takes care of the rest?

mmm, i'm gonna read the manual http://voxel.dl.sourceforge.net/project/quex/HISTORY/Ancient/Documentation/quex-doc-09y08m01d.pdf seems to be a great project.

I'm sending you my email on a private message to stop hijacking this thread  .

[Ceniza]

Why do you keep insisting in creating an overcomplicated const expression evaluator? Evaluating a const expression after macro expansion is straight forward using the grammar from the standard. I have pointed to an implementation multiple times, but it is always ignored. All you have to do is feed it the tokens while skipping whitespace (comments belong here too, if retained).

Proper macro expansion is complicated, though, even more when handling concatenation, "stringification" and recursive replacements. Think of something like this:

Code

#include <cstdio>

#define STR_HELPER(x) # x
#define STR(x) STR_HELPER(x)
#define CONCAT2(x, y) x ## y
#define CONCAT3(x, y, z) CONCAT2(x ## y, z)

int main()
{
    std::printf(STR($%@!&*));
    std::printf(STR(CONCAT3(this, /* COMMENT */ is, a /* another comment */ test)));
}

How many of you can actually tell me what the program is supposed to show on screen without compiling and running it? What if I replaced STR(x) to:

Code

#define STR(x) # x

Would it show the same?

[Ceniza]

There you go. The same code turned into something that should be easier to follow (it is, if you are not scared of pointers). All self contained, no templates. 4 tests included. Most of the conversion from the original code was made through find-and-replace. The grammar is included as documentation of each method.

[JGM]

So does this code works to evaluate macro expressions/conditions like these ones for example?

Code

#if VERBOSE >= 2
  print("trace message");
#endif

#if !defined(WIN32) || defined(__MINGW32__) 
...
#endif

I'm not sure If I'm using the correct terminology (sorry for that I'm a dumb xD) It would be easier to talk you on spanish xD, theres many parsing terminology I'm not familiar with

[JGM]

Well I documented and cleaned the code to some point here are the changes:

http://www.mediafire.com/?17skj2g70c86u50

Also I included a test case on the test folder, as my development environment is ubuntu/linux I created a shell script test/test.sh This script uses the debug binary of cpp_parser library and parses the test.hpp file also on test folder. (I took the example presented on this thread of #include on a typdef struct and made it part of the test case)

The original code on test.hpp is this one:

Code

#ifndef TEST_HPP
#define TEST_HPP

//Has multiply macro
#include "misc.h"

#ifndef MAX_VALUE
	#define MAX_VALUE 10000
#endif

#ifndef MAX_VALUE
	#define MAX_VALUE ShouldNotOccurre
#endif

typedef struct
{
	//The content of the struct in another file
	#include "object.h"
	//Should only be included once
	#include "object.h"
} Object;

namespace test
{
    int value = MAX_VALUE;
    int test = multiply;
	
    /**
     * Function one documentation
     * @return true otherwise false
     */
    bool function_one(const char &argument);

    /**
     * Function two documentation
     * @return The amount of characters found
     */
    unsigned int function_two(const char &argument);
};

//We undefine the MAX_VALUE macro
#undef MAX_VALUE

int value = MAX_VALUE;

#endif

and the cpp_parser binary returns this:

Code

 //Has multiply macro




typedef struct 
{ 
  //The content of the struct in another file


unsigned value; 
string test; 


  //Should only be included once
} Object; 

namespace test 
{ 
    int value = 10000; 
    int test = 4*5; 
  
    /**
     * Function one documentation
     * @return true otherwise false
     */                                                                             
    bool function_one(const char &argument); 

    /**
     * Function two documentation
     * @return The amount of characters found
     */                                                                                       
    unsigned int function_two(const char &argument); 
}; 

 //We undefine the MAX_VALUE macro

int value = MAX_VALUE; 

There are things left to implement and fix but for now basic functionality is working

[Ceniza]

So does this code works to evaluate macro expressions/conditions like these ones for example?

Code

#if VERBOSE >= 2
  print("trace message");
#endif

#if !defined(WIN32) || defined(__MINGW32__) 
...
#endif

I'm not sure If I'm using the correct terminology (sorry for that I'm a dumb xD) It would be easier to talk you on spanish xD, theres many parsing terminology I'm not familiar with

You need to tokenize and fully macro expand everything before feeding the evaluator.

For the first case you would need to expand VERBOSE to whatever its value is. Supposing it expands to '1', you would feed it:

Code

[ttNumber, "1"][ttWhiteSpace, " "][ttGreaterEqual, ">="][ttWhiteSpace, " "][ttNumber, "2"][ttEndOfTokens, ""]

For the second case you would need to expand defined(WIN32) and defined(__MINGW32__). Supposing both are defined, you would feed it:

Code

[ttNot, "!"][ttNumber, "1"][ttWhiteSpace, " "][ttOr, "||"][ttWhiteSpace, " "][ttNumber, "1"][ttEndOfTokens, ""]

Since it is a conditional (#if), all you care about is whether the result is 0 or not.

Instead of ttEndOfTokens as finalization, ttNewLine could be also added and handled just the same (must be added to the list of token types too).

If you want to learn more about the preprocessor in order to know what really needs to be implemented, check the C++0x draft chapters 2 (Lexical conventions) and 16 (Preprocessing directives) here.

[JGM]

...
You need to tokenize and fully macro expand everything before feeding the evaluator.

For the first case you would need to expand VERBOSE to whatever its value is. Supposing it expands to '1', you would feed it:

Code

[ttNumber, "1"][ttWhiteSpace, " "][ttGreaterEqual, ">="][ttWhiteSpace, " "][ttNumber, "2"][ttEndOfTokens, ""]

For the second case you would need to expand defined(WIN32) and defined(__MINGW32__). Supposing both are defined, you would feed it:

Code

[ttNot, "!"][ttNumber, "1"][ttWhiteSpace, " "][ttOr, "||"][ttWhiteSpace, " "][ttNumber, "1"][ttEndOfTokens, ""]

Since it is a conditional (#if), all you care about is whether the result is 0 or not.

Instead of ttEndOfTokens as finalization, ttNewLine could be also added and handled just the same (must be added to the list of token types too).

If you want to learn more about the preprocessor in order to know what really needs to be implemented, check the C++0x draft chapters 2 (Lexical conventions) and 16 (Preprocessing directives) here.

Thanks for your guidance! I need to fix some of the things to generate the preprocessed code in the exact line positions but without the macros for normal parsing (to get same line numbers and columns as original source code, not sure if it is so necessary). Also I created a parse expression function returning always true since I didn't had code to do that (planning to write it from scratch xD), so does the code you did is GPL (in other words can I use it xD)? Also I need to make macro values evaluation recursive (as you mentioned before) for cases like:

#define blah(x) x*2
#define blah2(y) blah(y)
#define blah3(z) blah2(z)

Other thing I need to implement is the conversion of expressions like x ## x or # x to a valid string or empty string since this data (I think) is not necessary for general parsing.

this is the code that actually handles the processing:

Code

for(unsigned int position=0; position<lines.size(); position++)
        {
            vector<preprocessor_token> tokens = lines[position];

            //Parse macro
            if(tokens[0].token == "#")
            {
                if(deepness == 0 || (deepness > 0 && last_condition_return[deepness]))
                {
                    if(tokens[1].token == "define")
                    {
                        define definition = parse_define(strip_macro_definition(tokens));
                        definition.file = file;
                        definition.line = tokens[2].line;
                        definition.column = tokens[2].column;
                        m_local_defines.push_back(definition);
                    }
                    if(tokens[1].token == "include")
                    {
                        string include_enclosure = tokens[2].token;
                        string include_file = "";
                        file_scope header_scope;

                        if(include_enclosure == "<")
                        {
                            for(unsigned int i=3; i<tokens.size(); i++)
                            {
                                if(tokens[i].token == ">")
                                {
                                    break;
                                }
                                else
                                {
                                    include_file += tokens[i].token;
                                }
                            }

                            m_headers_scope[include_file] = global;
                            header_scope = global;
                        }
                        else
                        {
                            for(unsigned int i=1; i<tokens[2].token.size(); i++)
                            {
                                if(tokens[2].token.at(i) == '"')
                                {
                                    break;
                                }
                                else
                                {
                                    include_file += tokens[2].token.at(i);
                                }
                            }

                            m_headers_scope[include_file] = local;
                            header_scope = local;
                        }

                        if(!is_header_parsed(include_file))
                        {
                            output += parse_file(include_file, header_scope); //To output the processed headers code
                            //parse_file(include_file, header_scope); //parses header without outputting

                            m_headers.push_back(include_file);
                        }
                    }
                    else if(tokens[1].token == "undef")
                    {
                        remove_define(tokens[2].token);
                    }
                    else if(tokens[1].token == "ifdef")
                    {
                        deepness++;
                        if(is_defined(tokens[2].token))
                        {
                            last_condition_return[deepness] = true;
                        }
                        else
                        {
                            last_condition_return[deepness] = false;
                        }
                    }
                    else if(tokens[1].token == "ifndef")
                    {
                        deepness++;
                        if(!is_defined(tokens[2].token))
                        {
                            last_condition_return[deepness] = true;
                        }
                        else
                        {
                            last_condition_return[deepness] = false;
                        }
                    }
                    else if(tokens[1].token == "if")
                    {
                        deepness++;
                        last_condition_return[deepness] = parse_expression(strip_macro_definition(tokens));
                    }
                }

                if(deepness > 0 && (tokens[1].token == "elif" || tokens[1].token == "else" || tokens[1].token == "endif"))
                {
                    if(tokens[1].token == "elif" && last_condition_return[deepness] != true)
                    {
                        last_condition_return[deepness] = parse_expression(strip_macro_definition(tokens));
                    }
                    else if(tokens[1].token == "else" && last_condition_return[deepness] != true)
                    {
                        last_condition_return[deepness] = true;
                    }
                    else if(tokens[1].token == "endif")
                    {
                        last_condition_return.erase(last_condition_return.find(deepness));
                        deepness--;
                    }
                }
            }

            //Parse code
            else
            {
                if(deepness == 0 || (deepness > 0 && last_condition_return[deepness]))
                {
                    unsigned int column = 1;

                    for(unsigned int i=0; i<tokens.size(); i++)
                    {
                        unsigned int columns_to_jump = tokens[i].column - column;

                        if(tokens[i].column <= 0)
                        {
                            columns_to_jump = 0;
                        }
                        else if(tokens[i].column < column)
                        {
                            columns_to_jump = column - tokens[i].column;
                        }

                        for(unsigned int y=0; y<columns_to_jump; y++)
                        {
                            output += " ";
                        }

                        if(tokens[i].type == identifier && is_defined(tokens[i].token))
                        {
                            output += get_define(tokens[i].token).value;
                        }
                        else
                        {
                            output += tokens[i].token;
                        }

                        column = tokens[i].column + tokens[i].token.size();
                    }

                    output += "\n";
                }
            }
        }

		return output;

as you can see, the code already handles nested macros correctly as basic ones (#define, #undef, #include, #ifdef, #ifndef) and with your code I would implement the  parse_expression function (like I said returns true by now, no evaluation) for #if and #elif evaluation. The tricky part is going to be recursiveness evaluation of macros.

I think I'm worrying to much about printing the parsed code with same lines since it is impossible on the case of multiple line macros like:

#define declare_table() class table{ \
int blah;\
};

Since these kind of macros are going to affect the line numbering on the output code.

My worry about same line positions was due to the fact of using the library also for refactoring, but a solution could be tough later I guess.

I will try to read the pages you mentioned of the standard draft xD (I bought an e-reader to accompany me on the nights xD)

Thanks again for your feedback!

Edit: Just did a quick look on c++ draft and I completly forgot about #line, #pragma, #error  :shock: but well I think these directives can be safely skipped except for #pragma that may include headers or things like that, what a pain  :lol:

Edit: Trigraph sequences - I knew about them but who would use that???  xD mmm I also forgot about Alternative tokens ,
uhhh also didnt tought about #include MACRO :S, well this post will remind me on things todo

[Ceniza]

... so does the code you did is GPL (in other words can I use it xD)?

It is not GPL, it is more like "do as you please, but if it kills your dog do not blame it on me" kind of license. I think it can be mixed with code under the GPL, but you better ask at least 3 lawyers to be sure

I would not recommend skipping #line as certain tools make use of it (mostly those that produce code, like lex/yacc or preprocessors themselves), and handling #error would be neat because you could inform the user about it way before hitting the 'build' button (as long as the files are always properly parsed to avoid false positives).

Keeping track of line numbers and files is, of course, extremely important. After all, the idea is for Code::Blocks to make use of it, and that information is vital. I think that making a list of the whole set of tools that want to be implemented, and what is needed for each one of them is the way to go to really know how fine grained line numbering needs to be stored.

[JGM]

It is not GPL, it is more like "do as you please, but if it kills your dog do not blame it on me" kind of license. I think it can be mixed with code under the GPL, but you better ask at least 3 lawyers to be sure

that's scary xD

I would not recommend skipping #line as certain tools make use of it (mostly those that produce code, like lex/yacc or preprocessors themselves), and handling #error would be neat because you could inform the user about it way before hitting the 'build' button (as long as the files are always properly parsed to avoid false positives).

Mmm so with #error the library should throw an exception.

Keeping track of line numbers and files is, of course, extremely important. After all, the idea is for Code::Blocks to make use of it, and that information is vital. I think that making a list of the whole set of tools that want to be implemented, and what is needed for each one of them is the way to go to really know how fine grained line numbering needs to be stored.

Well, for now when the code is first tokenized columns and line numbers are stored correctly, what I mean is when outputting the pre-processed code for full parsing of it (lexical analysis?).

The output code would need to be re-parsed with the issue of line numbers modified from original source, unless associations are made to previously tokenized original source.

Lets say we have this original code

Code

#include <something.h>
#define class_blah class test {\
char variable[50];\
};

class_blah

But the output of this would look different

Code

class something{
float test;
};

class test {
char variable[50];
};

It would parse as it should, but loosing original positions. We would still know on which files the class definitions were found but with incorrect line numbers and probably columns. My tiny brain can't think of a solution xD

[Ceniza]

Mmm so with #error the library should throw an exception.

Not necessarily. It could just store it somewhere for later retrieval. The parsing should continue in case it is a false positive.

Well, for now when the code is first tokenized columns and line numbers are stored correctly, what I mean is when outputting the pre-processed code for full parsing of it (lexical analysis?).

The output code would need to be re-parsed with the issue of line numbers modified from original source, unless associations are made to previously tokenized original source.

Lets say we have this original code

Code

#include <something.h>
#define class_blah class test {\
char variable[50];\
};

class_blah

But the output of this would look different

Code

class something{
float test;
};

class test {
char variable[50];
};

It would parse as it should, but loosing original positions. We would still know on which files the class definitions were found but with incorrect line numbers and probably columns. My tiny brain can't think of a solution xD

The preprocessing stage should output tokens, not text. The C++ parser's lexer job would be extremely simple: concatenate string literals into a single string literal token and turn numbers into either integral or floating-point tokens (flags may be needed to specify full type: unsigned, short, int, long, long long, float, double, long double). Identifiers could be turn into keywords here as well if not done before. Every other token would just pass through to the syntax analysis stage.

This is what the whole thing would, roughly, look like:

Preprocessor's Lexer -> Preprocessor -> Lexer -> Syntax analysis + symtab generation -> Semantic analysis.

Preprocessor's Lexer: Turns text into preprocessor tokens. Integral and floating-point values would be just "numbers". Keywords should be read as plain identifiers since the preprocessor does not care about them being a separate thing. File and line information is retrieved here.
Preprocessor: Resolves directives (#include, #if*, ...), discards tokens and builds new tokens when necessary (## and # operations). White spaces (space, newline, comments, ...) are, in theory, discarded as well.
Lexer: Converts "numbers" into proper tokens, concatenates contiguous string literals into a single string literal token and turns identifiers into keywords (the ones that are actually keywords, of course).
Syntax analysis: Checks that everything is properly "written" (class_decl ::= ttClass ttIdentifier ttSemiColon). An Abstract Syntax Tree can be built here, plus a symbols table.
Semantic analysis: Checks that everything makes sense: x = 3; // Is x a symbol in the current or a parent scope? Can it be assigned an integral type in any way (x is not const, x is integral, x has an overload of operator = that can be used, 3 can be turned into x's type and assigned, ...)?

That means some token types would not be seen by the preprocessor because its lexer would not produce them, most token types specifically for the preprocessor would have been consumed before reaching the lexer (at the next stage), and those few ones reaching it would be converted before being fed to the syntax analysis stage.

I hope it is clear enough, although its "roughness".

[JGM]

Not necessarily. It could just store it somewhere for later retrieval. The parsing should continue in case it is a false positive.

Yep, I thought that after implementing the ErrorException class

The preprocessing stage should output tokens, not text.

Ahhh true, and I kept thinking how to do it lol

[ollydbg]

This is what the whole thing would, roughly, look like:

Preprocessor's Lexer -> Preprocessor -> Lexer -> Syntax analysis + symtab generation -> Semantic analysis.

Preprocessor's Lexer: Turns text into preprocessor tokens. Integral and floating-point values would be just "numbers". Keywords should be read as plain identifiers since the preprocessor does not care about them being a separate thing. File and line information is retrieved here.
Preprocessor: Resolves directives (#include, #if*, ...), discards tokens and builds new tokens when necessary (## and # operations). White spaces (space, newline, comments, ...) are, in theory, discarded as well.
Lexer: Converts "numbers" into proper tokens, concatenates contiguous string literals into a single string literal token and turns identifiers into keywords (the ones that are actually keywords, of course).
Syntax analysis: Checks that everything is properly "written" (class_decl ::= ttClass ttIdentifier ttSemiColon). An Abstract Syntax Tree can be built here, plus a symbols table.
Semantic analysis: Checks that everything makes sense: x = 3; // Is x a symbol in the current or a parent scope? Can it be assigned an integral type in any way (x is not const, x is integral, x has an overload of operator = that can be used, 3 can be turned into x's type and assigned, ...)?

That means some token types would not be seen by the preprocessor because its lexer would not produce them, most token types specifically for the preprocessor would have been consumed before reaching the lexer (at the next stage), and those few ones reaching it would be converted before being fed to the syntax analysis stage.

I hope it is clear enough, although its "roughness".

very nice info.
But I think things get more  complex on parsing c++, because the c++ language is not context free, so Syntax analysis can not get the correct tree, because it need semantic information. So, we can not create a Bison grammar to parse c++ code, because both syntax and semantic should be combined.

and about the preprocessor side, checking an identifier (to see whether it is a keyword or a general variable name, function name) was really time consuming and context sensitive,  If we skip the expend the #include directive, we always get partial preprocessor result, so the macros may lost, and further #error kind message is also not correct.

how to avoid parsing a header file time from times? do we have a PCH like mechanism?

[Ceniza]

But I think things get more  complex on parsing c++, because the c++ language is not context free, so Syntax analysis can not get the correct tree, because it need semantic information. So, we can not create a Bison grammar to parse c++ code, because both syntax and semantic should be combined.

I think you are getting the job of the semantic analysis wrong. Let us say we have this code:

Code

float x = "a value";
++x;

You can build an AST from that, and the symtab will have that x is of type float. When you run the semantic analysis on that is when you will find that both lines have problems: assigning string literal to float, and pre-incrementing a float.

and about the preprocessor side, checking an identifier (to see whether it is a keyword or a general variable name, function name) was really time consuming and context sensitive,  If we skip the expend the #include directive, we always get partial preprocessor result, so the macros may lost, and further #error kind message is also not correct.

Right, I totally forgot to specify that.

During the preprocessing stage you need to build a macro replacements map, or whatever you want to call it. It would turn every identifier in a #define identifier as an "identifier to be replaced afterwards" or, simply, "macro". That map would be indexed by the identifier (the macro's name), and store the macro's type (plain macro or function-like macro), its parameters (for function-like macros), and the plain sequence of tokens that follow (the replacement). Have in mind that that sequence of tokens must NOT be macro expanded when stored.

When the preprocessor finds an identifier, it will search for it in the map. If it is found, build the list of parameters (each parameter being a sequence of non-expanded tokens) (in case it is a function-like macro), and proceed to do the replacement (expand it) over and over again until no more replacements are made. During this stage you need to keep a sort of call stack to properly handle what could otherwise become a recursive replacement (probably leading to an endless loop). Recursion is something the preprocessor must not do (check the standard).

how to avoid parsing a header file time from times? do we have a PCH like mechanism?

Well, you could store the result of preprocessing any file found through a #include. It would be indexed by the full file location, "sub-indexed" by the context plus dependencies, store the macro replacements map and the output (the final list of tokens).

The "sub-indexing" is important for a proper handling. The 'context plus dependencies' refers to all macros that were defined just before the file was #include'd, and their values. It is also important to know which other macros would cause the header to produce a different output (due to #if*). It is rather tricky to get it right, and it may cause the parsing to be a lot slower, although quite accurate. That is why, when programming, preprocessed headers should always be the first ones to be included (so they carry in as little context as possible).

In order to improve speed, as well as to simplify the implementation, the "sub-indexing" could be discarded. In other words: parse it once, store it like that, do not care about context. Handling multiple inclusion turns into the annoying part, though (as per this topic, you may want it, but, most of the time, you will not). [We are still on topic ]

"Stable" header files (like those that come with the compiler) should be parsed once and stored. You do not want to parse them every single time.

The last two paragraphs, as far as I know, is how the guys at Whole Tomato do it for Visual Assist X.

[ollydbg]

I think you are getting the job of the semantic analysis wrong. Let us say we have this code:

Code

float x = "a value";
++x;

You can build an AST from that, and the symtab will have that x is of type float. When you run the semantic analysis on that is when you will find that both lines have problems: assigning string literal to float, and pre-incrementing a float.

no, I have read some posts/threads on the web, look here:
7 Dealing with Ambiguities
and there are much more about parsing template instantiation code.
this always need semantic (type) information about the current identifier to support the syntax analysis.

[Ceniza]

The CDT parsers do not compute type information, the result being that some language constructs can be ambiguous, ...

According to that, they delay extracting type information until the semantic analysis stage, which is not practical for C/C++. That is completely unnecessary as the syntax analysis stage knows well what introduces new types. Since the symtab is populated as you build the AST, and you can also populate a "typetab", you can query information right away. It is, therefore, possible to know if x * y is a statement or an expression depending on whether or not x is in the symtab/"typetab". Otherwise, you will have to do the kind of trickery (ambiguity nodes) the CDT guys did.

Templates, on the other hand, require you to use the keyword typename to solve the ambiguity in favor of a statement, otherwise it is an expression.

[JGM]

Yay, I started working on the expression parser using ceniza code, what I'm struggling is on the macro expansion and recursivness of macro evaluations I'm still thinking whats the best way to do it.

@ollydbg
I copied the quex generated code on your repo to eliminate my custom tokenizer and use something that would simplify the process as improve it. Also I was studying the modifications you did on the tokenizer class. Is there a simpler example to follow? If not I guess I will have to study it more and search the documentation

[ollydbg]

@ollydbg
I copied the quex generated code on your repo to eliminate my custom tokenizer and use something that would simplify the process as improve it. Also I was studying the modifications you did on the tokenizer class. Is there a simpler example to follow? If not I guess I will have to study it more and search the documentation

I'm grad that you use quex generated parser now.
1, all the lexer code was under the folder: /cppparser/lexer, they were generated by the file cpp.qx(this is called lexer grammar file), and the running command to generate is "cpp.bat" (windows command to call quex library). If you just want to use the generated lexer, you do not need to install python and quex, because the generated code has contains all the files necessary to compile. If you want to modify the cpp.qx file, then you need to install python and quex.

2, I use the mode "pointing to a buffer" in the generated lexer. which is, when it initialized, I set a NULL pointer to the lexer.

Code

m_Quex((QUEX_TYPE_CHARACTER*)s_QuexBuffer,4,(QUEX_TYPE_CHARACTER*)s_QuexBuffer+1)

this code is the initialization of the lexer, and s_QuexBuffer is infact a buffer of NULL.

Code

QUEX_TYPE_CHARACTER Tokenizer::s_QuexBuffer[4] = {0,0,0,0};

3, When you own buffer is ready, I just "point to" it, see:

Code

bool Tokenizer::ReadFile()
{
    bool success = false;
    cc_string fileName = cc_text("");
    if (m_pLoader)
    {
        fileName = m_pLoader->fileName();

        const char * pBuffer = m_pLoader->data();
        m_BufferLen = m_pLoader->length();

        if( m_BufferLen != 0)
            success = true;


        m_Quex.reset_buffer((QUEX_TYPE_CHARACTER*)pBuffer,
                                       m_BufferLen+2,
                                       (QUEX_TYPE_CHARACTER*)pBuffer+m_BufferLen+1);

        (void)m_Quex.token_p_switch(&m_TokenBuffer[0]);

        cout<< "set buffer size" << (int)QUEX_SETTING_BUFFER_SIZE <<endl;

        return true;

}

the char buffer is loaded by "m_pLoader" which is a file loader. so, it contains two info, one is its buffer start address, the other is the length

Code

        const char * pBuffer = m_pLoader->data();
        m_BufferLen = m_pLoader->length();

Code

&m_TokenBuffer[0]

this is the Token address set by the user, so when you call

Code

(void)m_Quex.token_p_switch(&m_TokenBuffer[0]);

this will let the lexer go one step and fill the Token.

4, my Tokenizer is modified from the Tokenizer class in CC's current implementation, but has a lot of things changed. The normal way to receive the Token is like below:

Code

bool Tokenizer::FetchToken(RawToken * pToken)
{
    (void)m_Quex.token_p_switch(pToken);

    QUEX_TYPE_TOKEN_ID id = m_Quex.receive();

    if( id == TKN_TERMINATION )
    {
       m_IsEOF = true;
       return false;
    }
    return true;
}

you supply a token address, and the lexer fill the token. then, you can get the Token's Id and Token's text(if it is an identifier), also it's line and column information.

If you have some problems using quex, feel free to ask me.

[JGM]

after writing a simple function for evaluation of expressions I got stuck  

This is the test case:

Code

#ifndef MAX_VALUE
	#define MAX_VALUE 10000
#endif

#ifndef MAX_VALUE
	#define MAX_VALUE ShouldNotOccurre
#endif

#if MAX_VALUE > 100 //This is the simple test I was to evaluate
	int testy;
#endif

Here the function I wrote to prepare a list of Token for the constexprevaluator class

Code

/**
         * Converts an expression from a #if, #else, etc to an array of elements with macros expanded
         * @return Vector with tokens that can be used to evalulate the expression by the ConstExprEvaluator class.
         */
const vector<Token> preprocessor::expand_macro_expression(const vector<preprocessor_token> &expression)
{
	    vector<Token> tokens;

	    for(unsigned int i=0; i<expression.size(); i++)
        {
            Token token_to_add;

            if(expression[i].type == identifier)
            {
                token_to_add.type = ttNumber;

                if(is_defined(expression[i].token))
                {
                    define macro = get_define(expression[i].token);

                    //For macro definitions
                    if(macro.parameters.size() <= 0)
                    {
                        if(macro.value == "")
                        {
                            //The macro is defined but without a predifined value for it so we default to 1
                            token_to_add.value = "1";
                        }
                        else
                        {
                            //Tha macro has a predifined value (We should check if it's a valid number)
                            token_to_add.value = macro.value;
                        }
                    }

                    //For macro functions, we need to parse the parameters and then evaluate with recursive function
                    else
                    {
                        //for now we just return 1 but this is totally wrong
                        token_to_add.value = "1";
                    }
                }
                else
                {
                    token_to_add.value = "0";
                }
            }

            //Adds any numbers found
            else if(expression[i].type == number)
            {
                token_to_add.type = ttNumber;
                token_to_add.value = expression[i].token;
            }

            //Handle any other operator found
            else
            {
                if(expression[i].token == "?")
                {
                    token_to_add.type = ttQuestion;
                    token_to_add.value = "?";
                }
                else if(expression[i].token == ":")
                {
                    token_to_add.type = ttColon;
                    token_to_add.value = ":";
                }
                else if(expression[i].token == "||")
                {
                    token_to_add.type = ttOr;
                    token_to_add.value = "||";
                }
                else if(expression[i].token == "&&")
                {
                    token_to_add.type = ttAnd;
                    token_to_add.value = "&&";
                }
                else if(expression[i].token == "|")
                {
                    token_to_add.type = ttBitOr;
                    token_to_add.value = "|";
                }
                else if(expression[i].token == "^")
                {
                    token_to_add.type = ttBitXOr;
                    token_to_add.value = "^";
                }
                else if(expression[i].token == "&")
                {
                    token_to_add.type = ttBitAnd;
                    token_to_add.value = "&";
                }
                else if(expression[i].token == "==")
                {
                    token_to_add.type = ttEqual;
                    token_to_add.value = "==";
                }
                else if(expression[i].token == "!=")
                {
                    token_to_add.type = ttNotEqual;
                    token_to_add.value = "!=";
                }
                else if(expression[i].token == "<")
                {
                    token_to_add.type = ttLess;
                    token_to_add.value = "<";
                }
                else if(expression[i].token == ">")
                {
                    token_to_add.type = ttGreater;
                    token_to_add.value = ">";
                }
                else if(expression[i].token == "<=")
                {
                    token_to_add.type = ttLessEqual;
                    token_to_add.value = "<=";
                }
                else if(expression[i].token == ">=")
                {
                    token_to_add.type = ttGreaterEqual;
                    token_to_add.value = ">=";
                }
                else if(expression[i].token == "<<")
                {
                    token_to_add.type = ttLShift;
                    token_to_add.value = "<<";
                }
                else if(expression[i].token == ">>")
                {
                    token_to_add.type = ttRShift;
                    token_to_add.value = ">>";
                }
                else if(expression[i].token == "+")
                {
                    token_to_add.type = ttPlus;
                    token_to_add.value = "+";
                }
                else if(expression[i].token == "-")
                {
                    token_to_add.type = ttMinus;
                    token_to_add.value = "-";
                }
                else if(expression[i].token == "*")
                {
                    token_to_add.type = ttTimes;
                    token_to_add.value = "*";
                }
                else if(expression[i].token == "/")
                {
                    token_to_add.type = ttDivide;
                    token_to_add.value = "/";
                }
                else if(expression[i].token == "%")
                {
                    token_to_add.type = ttModulo;
                    token_to_add.value = "%";
                }
                else if(expression[i].token == "!")
                {
                    token_to_add.type = ttNot;
                    token_to_add.value = "?";
                }
                else if(expression[i].token == "~")
                {
                    token_to_add.type = ttBitNeg;
                    token_to_add.value = "~";
                }
                else if(expression[i].token == "(")
                {
                    token_to_add.type = ttLParen;
                    token_to_add.value = "(";
                }
                else if(expression[i].token == ")")
                {
                    token_to_add.type = ttRParen;
                    token_to_add.value = ")";
                }
            }

            tokens.push_back(token_to_add);
        }

        Token endToken = {ttEndOfTokens, ""};
        tokens.push_back(endToken);


        return tokens;
}

And this is the function that catches the tokens and pass them to the evaluator:

Code

/**
         * Evaluates a macro expression/condition
         * @param define_declaration A macro object
         * @return true if condition is true (duh!) false otherwise
         */
const bool preprocessor::parse_expression(const vector<preprocessor_token> &expression)
	{
	    bool return_value = false;

	    const vector<Token> tokens_vector = expand_macro_expression(expression);

        Token tokens[tokens_vector.size()];

	    for(unsigned int i=0; i<tokens_vector.size(); i++)
        {
            tokens[i] = tokens_vector[i];

            cout << tokens_vector[i].value; //To check the tokens are correct (debugging)
        }

        try
        {
            PCToken pcToken = &tokens[0];

            if(ConstExprEvaluator::eval(&pcToken) > 0)
            {
                return_value = true;
            }
        }
        catch (const PreprocessorError &prepError)
        {
            return_value = false;

            //TODO add this exception to m_error
            std::cerr << "Exception: " << prepError.getMessage() << "\n";
        }

        return return_value;
	}

and I'm getting the following exception: Exception: Error parsing constant-expression at token

@ceniza
whats the exact meaning of that exception? and sorry for my noobness  :oops:

Edit: The expression to evalualte is 10000 > 100

Edit #2: After drinking a glass of water I was thinking on the code I wrote and I figured out I was adding empty tokens on the for loop sorry for that one!

[JGM]

If you have some problems using quex, feel free to ask me.

Thanks ollydbg!, everything is clearer now I copied the right files as you mentioned from the lexer dir, but I didn't understand the logic, now it's clearer

[Ceniza]

@ceniza
whats the exact meaning of that exception? and sorry for my noobness  :oops:

Edit: The expression to evalualte is 10000 > 100

Edit #2: After drinking a glass of water I was thinking on the code I wrote and I figured out I was adding empty tokens on the for loop sorry for that one!

I guess Edit #2 means that you have solved the issue already.

A few comments:

* If a macro is defined as nothing (empty), the replacement is empty, not "1". I hope it is a placeholder.
* There is no need to adapt your code to use the ConstExprEvaluator's interface. Instead, adapt ConstExprEvaluator's to your code (the current interface was a quick hack to remove the one that used TokenProvider).
* A constant expression is true when it is not 0 (!= 0), not when it is positive (> 0).
* The code of the ConstExprEvaluator unnecessarily calls the peek function over and over again. It was a quick replacement to get it working. Calling skipWhiteSpace on eval() once and using the current token's value instead of calling peek is more optimal. Additionally, replace the usage of operator ++ by a call to a function that advances the 'iterator' (using operator ++) followed by a call to skipWhiteSpace (call it nextToken).

[JGM]

I guess Edit #2 means that you have solved the issue already.

Yep it is working now  :wink: Your expression evaluator code is really cool!

* If a macro is defined as nothing (empty), the replacement is empty, not "1". I hope it is a placeholder.

ohhh

* A constant expression is true when it is not 0 (!= 0), not when it is positive (> 0).

In other words, negative numbers evaluate to true, didn't supposed that

* The code of the ConstExprEvaluator unnecessarily calls the peek function over and over again. It was a quick replacement to get it working. Calling skipWhiteSpace on eval() once and using the current token's value instead of calling peek is more optimal. Additionally, replace the usage of operator ++ by a call to a function that advances the 'iterator' (using operator ++) followed by a call to skipWhiteSpace (call it nextToken).

There are also many things on my code that need optimization

Thanks for all your feedback, great source of reference 

[ollydbg]

find some preprocessor source links
https://sourceforge.net/apps/trac/cppcheck/ticket/516
PS: cppcheck has a preprocessor too.