Is it possible for the parser to support newlib prototypes?

[WinterMute]

Newlib function prototypes take the form

Code

FILE *	_EXFUN(fopen, (const char *__restrict _name, const char *__restrict _type));

which obviously isn't picked up by the codeblocks parser as a function prototype.

Is it possible to get the cb parser to interpret these prototypes and add them to the code completion list?

To clarify - the symbol browser shows many _EXFUN( ...  ) prototypes but the _EXFUN macro is just there to adjust the prototypes for various platforms. For instance on CYGWIN it's

Code

#define	_EXFUN(name, proto)		__cdecl name proto

but for other platforms it's

Code

#define	_EXFUN(name, proto)		name proto

Thus the above example for fopen ends up as

Code

FILE * fopen (const char *__restrict _name, const char *__restrict _type);

It would be nice if it was possible to somehow get the parser to expand that particular macro, even if it's just a regex somewhere. I don't really expect it to be able to follow the newlib headers & pick the right macro to expand.

[ollydbg]

Yes, it is possible, see my patch(a hack)

Code

 src/plugins/codecompletion/parser/tokenizer.cpp | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/src/plugins/codecompletion/parser/tokenizer.cpp b/src/plugins/codecompletion/parser/tokenizer.cpp
index 2176ab8..fee843d 100644
--- a/src/plugins/codecompletion/parser/tokenizer.cpp
+++ b/src/plugins/codecompletion/parser/tokenizer.cpp
@@ -1236,6 +1236,9 @@ void Tokenizer::ReplaceMacro(wxString& str)
                 }
             }
         }
+        // if in macro expansion mode, we don't let the manually replacement rule
+        // executed again, so just returned
+        return;
     }
 
     wxStringHashMap::const_iterator it = s_Replacements.find(str);
@@ -1296,6 +1299,15 @@ void Tokenizer::ReplaceMacro(wxString& str)
             str = DoGetToken();
         }
     }
+    else if (it->second[0] == _T('@'))
+    {
+        // trigger the macro replacement, so that we can look up the keyword in TokenTree for a
+        // macro usage, note that ReplaceBufferText() below just move backword the index, but
+        // don't change the buffer
+        if(ReplaceBufferText(it->first, false))
+            str = DoGetToken();
+
+    }
     else // for other user defined rules, just do a buffer content replacement
     {
         if (it->second != str && ReplaceBufferText(it->second, false))
@@ -1686,7 +1698,8 @@ void Tokenizer::SplitArguments(wxArrayString& results)
         else if (token == _T(")"))
             --level;
 
-        if (token == _T(","))
+        // comma is a delimit only it is not wrapper by ()
+        if (token == _T(",") && level == 1)
         {
             results.Add(piece);
             piece.Clear();

You need to add a replacement rule in CC setting: _EXFUN  -> @

And the result can be shown in the screen shot below:

[ollydbg]

In trunk now.

Code

Revision: 9829
Author: ollydbg
Date: 2014-6-24 10:48:08
Message:
* CC: add a new CC replacement rule: XXXXX -> @, and document the ReplaceMacro() function. The new rule triggers the tokenizer to switch to macro replacement mode, thus a macro definition can be looked up in the token tree. This fixes a bug report here: http://forums.codeblocks.org/index.php/topic,19278.0.html.
-------------------------------
M(T ) : /trunk/src/plugins/codecompletion/parser/tokenizer.cpp

M(T ) : /trunk/src/plugins/codecompletion/parser/tokenizer.h

I'm also updating the wiki page by adding this rule:
http://wiki.codeblocks.org/index.php?title=Code_Completion_Design

[Teybeo]

Does this patch enables support for glew.h functions definitions ?

Example for the OpenGL function

Code

glMultiDrawElements(GLenum mode, const GLsizei *count, GLenum type, const GLvoid **indices, GLsizei drawcount);

Code

typedef void (GLAPIENTRY * PFNGLMULTIDRAWELEMENTSPROC) (GLenum mode, const GLsizei *count, GLenum type, const GLvoid **indices, GLsizei drawcount);
...
GLEW_FUN_EXPORT PFNGLMULTIDRAWELEMENTSPROC __glewMultiDrawElements;
...
#define glMultiDrawElements GLEW_GET_FUN(__glewMultiDrawElements)


#define GLEW_FUN_EXPORT GLEWAPI
#define GLEW_GET_FUN(x) x

#ifdef GLEW_STATIC
#  define GLEWAPI extern
#else
#  ifdef GLEW_BUILD
#    define GLEWAPI extern __declspec(dllexport)
#  else
#    define GLEWAPI extern __declspec(dllimport)
#  endif
#endif

[ollydbg]

Does this patch enables support for glew.h functions definitions ?

Example for the OpenGL function

Code

glMultiDrawElements(GLenum mode, const GLsizei *count, GLenum type, const GLvoid **indices, GLsizei drawcount);

This is a function pointer, not a function declaration, right? EDIT, true, that this is a function pointer, but we can call it like a normal function call like glMultiDrawElements(...), see Function Pointers in C and C++ - Cprogramming.com

I just tested, and currently, CC only regard the glMultiDrawElements as a macro definition. What is the expect behavior do you want?

EDIT:
The correct test code could be:

Code

#ifdef GLEW_STATIC
#  define GLEWAPI extern
#else
#  ifdef GLEW_BUILD
#    define GLEWAPI extern __declspec(dllexport)
#  else
#    define GLEWAPI extern __declspec(dllimport)
#  endif
#endif

#define GLEW_FUN_EXPORT GLEWAPI
#define GLEW_GET_FUN(x) x

typedef void (GLAPIENTRY * PFNGLMULTIDRAWELEMENTSPROC) (GLenum mode, const GLsizei *count, GLenum type, const GLvoid **indices, GLsizei drawcount);

GLEW_FUN_EXPORT PFNGLMULTIDRAWELEMENTSPROC __glewMultiDrawElements;

#define glMultiDrawElements GLEW_GET_FUN(__glewMultiDrawElements)

glMultiDrawElements

Or even simpiler

Code

#define GLEW_FUN_EXPORT extern
#define GLEW_GET_FUN(x) x

typedef void ( * PFNGLMULTIDRAWELEMENTSPROC) (int a);

GLEW_FUN_EXPORT PFNGLMULTIDRAWELEMENTSPROC __glewMultiDrawElements;

#define glMultiDrawElements GLEW_GET_FUN(__glewMultiDrawElements)

glMultiDrawElements

[Huki]

Today I was about to submit my patch for function calltips, and coincidentally found this post. My patch adds calltips support for some complex macro functions, and variables of function pointer typedefs. Both are required to support the glew functions.

1) Advanced macro functions: supported cases,
- Preprocosser token defined to function name, macro name or variable of typedef'd function pointer.
- Nested cases of the above (defines and macro calls are expanded).

Code

void FUNC(...);
#define DEFINE FUNC
DEFINE( // show calltip for FUNC()

#define MACRO(..) ...
#define DEFINE MACRO
DEFINE( // show calltip for MACRO()
MACRO( // same

#define WRAPPER(_x) FUNC
#define DEFINE WRAPPER(x)
DEFINE( // show calltip for FUNC()

2) Fixed the support for typedef'd function pointer (there was already some non-working code for it).

Code

typedef void (* t_pfnMyProc) (int a);
t_pfnMyProc myProc;
myProc( // show calltip

Also combinations of 1) and 2) are supported (like your glew test code).

3) Some improvement and notes for GetMacroExpendedText() which is used for above features.

I'm attaching the patch file.

[ollydbg]

I'm sleep now, will look into it tomorrow, thanks.

[ollydbg]

I try to test this code with your patch:

Code

void my_f(int b);

void (*foo)(int a);

foo = &my_f;

foo(5);

But I don't have tip info if the mouse hover on the "foo(5)".
Find declaration of "foo(5)" gives nothing.

[Huki]

I try to test this code with your patch:

Code

void my_f(int b);

void (*foo)(int a);

foo = &my_f;

foo(5);

But I don't have tip info if the mouse hover on the "foo(5)".
Find declaration of "foo(5)" gives nothing.

I think normal function pointer variables parsing require another patch (I will send it next, but it requires some cleanup before). This patch supports typedef'd function pointers only (and it's a patch for function calltips, not tooltips).
Try this:

Code

void my_f(int b);

typedef void (*t_foo)(int a);
t_foo foo;

foo = my_f; // I think we should assign function directly, not &my_f
            // EDIT: Ok, in fact both my_f and &my_f are correct...

foo( // shows calltip for void t_foo(int a)

[ollydbg]

Hi, Huki, thanks for the explanation, when testing and reviewing your patch, I see this comment which I'm not clear:

Code

                    // NOTE: ReplaceFunctionLikeMacro will recursively expand all defines and macro calls,
                    // but will not expand macro names, which is what we want
                    smallTokenizer.ReplaceFunctionLikeMacro(tk);
                    tk = tree->at(tree->TokenExists(smallTokenizer.GetToken(), -1, tkFunction|tkMacroDef|tkVariable));
                    if (tk && smallTokenizer.PeekToken().empty()) // only if the expanded result is a single token
                        token = tk;

What does the "will not expand macro names" in comments means? as I see, ReplaceFunctionLikeMacro() did a full expansion until nothing is expanded.

[Huki]

Hi, Huki, thanks for the explanation, when testing and reviewing your patch, I see this comment which I'm not clear:

Code

                    // NOTE: ReplaceFunctionLikeMacro will recursively expand all defines and macro calls,
                    // but will not expand macro names, which is what we want
                    smallTokenizer.ReplaceFunctionLikeMacro(tk);
                    tk = tree->at(tree->TokenExists(smallTokenizer.GetToken(), -1, tkFunction|tkMacroDef|tkVariable));
                    if (tk && smallTokenizer.PeekToken().empty()) // only if the expanded result is a single token
                        token = tk;

What does the "will not expand macro names" in comments means? as I see, ReplaceFunctionLikeMacro() did a full expansion until nothing is expanded.

Hi,
Consider a macro definition like this:

Code

#define MACRO(_x) ...

It must be used like this (this is a macro call):

Code

MACRO(5);

Now we have a define pointing to this macro:

Code

#define DEFINE MACRO

Note that there is no "()" after MACRO.. so this is not a valid "macro call", just the macro name is used alone. In normal code this is illegal and probably an error, but when used like this in the define, it means the preprocessor token DEFINE should be used like a macro:

Code

DEFINE(5); // same as MACRO(5);

So to qualify as a macro name, the macro definition should have arguments (or empty parentheses), but the macro usage should be without any parentheses.

In ReplaceFunctionLikeMacro() I have added this test to make sure such macro names don't get expanded (expanding them is illegal anyway).

Code

@@ -1954,16 +1962,26 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
[...]
+    // don't replace anything if the arguments are missing
+    if (!SplitArguments(actualArgs))
         return false;

BTW: valid preprocessor tokens will still be expanded (i.e., ordinary preprocessor tokens that have no parentheses in the definition itself, like #define DEFINE ...
I added that feature too in the patch:

Code

@@ -1937,6 +1938,13 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
     if (tk->m_FullType.Find(tk->m_Name) != wxNOT_FOUND)
         return false;
 
+    // if not even "()" is found [in the definition] then it's a normal preprocessor define, so handle as such
+    if (tk->m_Args.IsEmpty())
+    {
+        expandedText = tk->m_FullType;
+        return true;    // return true for ReplaceBufferText()
+    }
+

Also valid macro calls will get expanded:

Code

#define GLEW_GET_FUN(x) x

// now
#define glMultiDrawElements GLEW_GET_FUN(__glewMultiDrawElements)
// becomes
#define glMultiDrawElements __glewMultiDrawElements

// show calltip for __glewMultiDrawElements(...), which can be 
// a function, macro, function ptr or typedef'd function ptr
// (currently function ptr are not parsed properly in DoParse() so 
//  it's not supported for now. But typedef'd func ptrs are supported)
glMultiDrawElements(

Likewise we have function calls and function names, see some example code:

Code

// MessageBoxA is a function name
// it means we can call MessageBox() and get calltip for MessageBoxA()
#define MessageBox MessageBoxA

// MessageBoxA is a function CALL
// it means we cannot use MessageBox like a call, it's just a define.
#define MessageBox MessageBoxA(...)

[ollydbg]

Hi,
Consider a macro definition like this:

Code

#define MACRO(_x) ...

It must be used like this (this is a macro call):

Code

MACRO(5);

Now we have a define pointing to this macro:

Code

#define DEFINE MACRO

Note that there is no "()" after MACRO.. so this is not a valid "macro call", just the macro name is used alone. In normal code this is illegal and probably an error,

Thanks for the explain.
It looks like this code above is valid, and I just create a simple test code, and it build fine under MinGW G++

Code

#define MACRO(_x) _x
#define DEFINE MACRO

int main()
{
    int x;
    DEFINE(x);
    return 0;
}

but when used like this in the define, it means the preprocessor token DEFINE should be used like a macro:

Code

DEFINE(5); // same as MACRO(5);

So to qualify as a macro name, the macro definition should have arguments (or empty parentheses), but the macro usage should be without any parentheses.

What does the last sentence means? I don't get the idea.

In ReplaceFunctionLikeMacro() I have added this test to make sure such macro names don't get expanded (expanding them is illegal anyway).

Sorry, I still not quite understand this sentence. (Maybe, dur to my poor English.....)

Code

@@ -1954,16 +1962,26 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
[...]
+    // don't replace anything if the arguments are missing
+    if (!SplitArguments(actualArgs))
         return false;

I see that the only usage of the GetMacroExpendedText() is here: (BTW: typo here? should be "expanded text"?)

Code

bool Tokenizer::ReplaceFunctionLikeMacro(const Token* tk, bool updatePeekToken)
{
    wxString macroExpandedText;
    if ( GetMacroExpendedText(tk, macroExpandedText) )
        return ReplaceBufferText(macroExpandedText, updatePeekToken);
    return false;
}

But as I see that the function ReplaceFunctionLikeMacro is called in some condition that tk->m_Args is not empty. It looks like m_Args is used to distinguish whether a macro definition is variable-like or function-like, but if I realize that even function-like macro definitions are allowed to use empty formal argument list, so the condition is not true any more.

GetMacroExpendedText did some tricks, it just put the formal arguments before actual arguments, so the buffer becomes

Code

.....  ( formal arguments )  ( actual arguments ) ....
       ^ --- m_TokenIndex

Oh, so for a function-like macro definition which has empty argument list, the m_Args is "()".  SplitArguments(actualArgs) function will still return true, but both the actualArgs  and the formalArgs are empty,  in this case, we can directly return the macro's definition string without any replacement.

BTW: it looks like ReplaceFunctionLikeMacro() only runs one level replacement, if you have several replacement rules (A -> B ->C), call this function only make a (A->B), but I think (B->C) will be happens after some GetToken() call. So, it looks like our macro expansion is not similar as the C language standard.

BTW: valid preprocessor tokens will still be expanded (i.e., ordinary preprocessor tokens that have no parentheses in the definition itself, like #define DEFINE ...
I added that feature too in the patch:

Code

@@ -1937,6 +1938,13 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
     if (tk->m_FullType.Find(tk->m_Name) != wxNOT_FOUND)
         return false;
 
+    // if not even "()" is found [in the definition] then it's a normal preprocessor define, so handle as such
+    if (tk->m_Args.IsEmpty())
+    {
+        expandedText = tk->m_FullType;
+        return true;    // return true for ReplaceBufferText()
+    }
+

Yes, this is some kind of expanding variable-like macro.

Edit: I think I still need time to review the CC code and your patches.

[ollydbg]

Some test:
Here is the test code

Code

#define AAA BBB
#define BBB CCC
#define CCC DDD
#define DDD EEE

int AAA;

Nornally, when parsed, you get a variable Token named "AAA", since CC don't check every token for macros, so AAA does not trigger macro replacement.

If you create a user replacement rule by adding AAA -> @ in the CC's setting(this just move back the m_TokenIndex, and put the Tokenizer in macro replacement mode), you will get a variable Token named "EEE", which is correct.

I see that DoGetToken() can be recurively called, thus recursive macro replacement happens when DoGetToken() is called.

EDIT:
In your patch

Code

                    //NOTE: ReplaceFunctionLikeMacro will recursively expand all defines and macro calls,
                    // but will not expand macro names, which is what we want
                    smallTokenizer.ReplaceFunctionLikeMacro(tk);

I think the comment is not correct right? Since ReplacefunctionLikeMacro just do text replacement once.

EDIT2:
If I change to this code

Code

#define AAA BBB
#define BBB CCC
#define CCC DDD
#define DDD EEE
#define EEE BBB

int AAA;

You will quickly get an infinite loop, until we run to the limit:

Code

    if (m_RepeatReplaceCount > 0)
    {
        if (m_RepeatReplaceCount >= s_MaxRepeatReplaceCount)
        {
            m_TokenIndex = m_BufferLen - m_FirstRemainingLength;
            m_PeekAvailable = false;

To solve this issue, I think
1, collecting the used macros, and don't use it again
2, expand all the macros once, not by recursive call of DoGetToken

[ollydbg]

EDIT:
In your patch

Code

                    //NOTE: ReplaceFunctionLikeMacro will recursively expand all defines and macro calls,
                    // but will not expand macro names, which is what we want
                    smallTokenizer.ReplaceFunctionLikeMacro(tk);

I see that when you call ReplaceFunctionLikeMacro(), then the Tokenizer goes to the macro replacement mode, so the next time you call smallTokenizer.GetToken(), macro replacement will recursively expand all.

[Huki]

Oh, sorry about the confusion. Regarding macro expansion: for any macro, we have 1) the macro definition, and 2) the macro usage. Each of them can be of two kinds: 1) having parentheses (...), or 2) not having parentheses. So you can see there are totally 4 cases to handle. Then the macro can either be simple, or nested (needs recursive expansion). I will try to explain the different cases one by one.

Case 1: Both macro definition and macro usage do not have (). Eg,

Code

#define FIVE 5
int a = FIVE;

We can call this a variable-like macro define / usage.

Case 2: Both macro definition and macro usage have (). Eg,

Code

#define MUL(_x) (_x * 5)
int a = MUL(1);

We can call this a function-like macro define / usage.

Case 3: Macro definition does not have () but macro usage does have it. Eg,

Code

#define MUL(_x) (_x * 5)

#define OPER MUL

// OPER(1) is an example of Case 3 macro usage
int a = OPER(1);

Here, OPER is defined as a variable-macro, but used as a function-macro. We should expand it like a variable-macro using plain ReplaceBufferText(). In my patch I have added support for this case (the test added at the beginning of GetMacroExpendedText()).

Case 4: Macro definition has the () but macro usage does not have it. Eg,

Code

#define MUL(_x) (_x * 5)

// MUL is an example of Case 4 macro usage
#define OPER MUL // this line is valid (but CC should not expand MUL)

// these two lines are valid
int a = OPER(1);
int b = MUL(1);

int c = MUL; // this line is invalid (CC should not expand MUL)

Here, MUL is defined as a function-like macro. But in the line "#define OPER MUL" it is used without parentheses. This means our CC engine should treat it like an error and leave it alone (do not expand it). We can use this behavior to stop our recursive expansion at this point, and use the final result "MUL" for the calltips.

It looks like this code above is valid, and I just create a simple test code, and it build fine under MinGW G++
[...]

Yes, this is valid, but still we should not expand it (we need to stop there and use the MUL token for our calltips). Imagine if we expand it:

Code

#define OPER MUL
// becomes
#define OPER (_x * 5) // invalid because OPER cannot accept any arguments!

So generally our macro expansion code should always treat Case 4 like an error and don't expand it...

That is the reason for this test added:

Code

@@ -1954,16 +1962,26 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
[...]
+    // don't replace anything if the [actual (or usage)] arguments are missing
+    if (!SplitArguments(actualArgs))
         return false;

I hope that is clear now.

I see that the only usage of the GetMacroExpendedText() is here: (BTW: typo here? should be "expanded text"?)

Yes, here is the only usage of this function. GetMacroExpendedText() should return "true" if we want ReplaceBufferText(), and "false" if we want to leave the macro alone and don't replace. But even if there is empty () in the macro definition we have to handle it in GetMacroExpendedText() before returning true.
And yes, it should be "expanded text".

It looks like m_Args is used to distinguish whether a macro definition is variable-like or function-like, but if I realize that even function-like macro definitions are allowed to use empty formal argument list, so the condition is not true any more.

Yes, but m_Args is not empty even if there is empty parentheses. So we can safely check for m_Args->IsEmpty(). A function-macro like this: "#define MACRO() ..." will have in m_Args: "()". Only for variable-macro the m_Args is really empty.

BTW: it looks like ReplaceFunctionLikeMacro() only runs one level replacement, if you have several replacement rules (A -> B ->C), call this function only make a (A->B), but I think (B->C) will be happens after some GetToken() call.

I think the comment is not correct right? Since ReplacefunctionLikeMacro just do text replacement once.

Well, yes, the actual recursive expansion happens in the next GetToken() call, but I wanted to keep the comments simple and easy to read. Technically, ReplacefunctionLikeMacro() expands one level, and turns on the flag for further expansion in the next GetToken() or PeekToken().
In my patch there is:

Code

+                    smallTokenizer.ReplaceFunctionLikeMacro(tk);
+                    tk = tree->at(tree->TokenExists(smallTokenizer.GetToken(), ...

So, full recursive expansion occurs in the GetToken(), but when we hit Case 4, GetMacroExpendedText() will return false, and it won't be expanded further. We use that result for the calltips.