Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => CodeCompletion redesign => Topic started by: WinterMute on May 16, 2014, 01:44:47 pm

Title: Is it possible for the parser to support newlib prototypes?
Post by: WinterMute on May 16, 2014, 01:44:47 pm
Newlib function prototypes take the form
Code
FILE *	_EXFUN(fopen, (const char *__restrict _name, const char *__restrict _type));
which obviously isn't picked up by the codeblocks parser as a function prototype.

Is it possible to get the cb parser to interpret these prototypes and add them to the code completion list?

To clarify - the symbol browser shows many _EXFUN( ...  ) prototypes but the _EXFUN macro is just there to adjust the prototypes for various platforms. For instance on CYGWIN it's
Code
#define	_EXFUN(name, proto)		__cdecl name proto
but for other platforms it's
Code
#define	_EXFUN(name, proto)		name proto
Thus the above example for fopen ends up as
Code
FILE * fopen (const char *__restrict _name, const char *__restrict _type);

It would be nice if it was possible to somehow get the parser to expand that particular macro, even if it's just a regex somewhere. I don't really expect it to be able to follow the newlib headers & pick the right macro to expand.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on June 20, 2014, 02:32:19 am
Yes, it is possible, see my patch(a hack)
Code
 src/plugins/codecompletion/parser/tokenizer.cpp | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/src/plugins/codecompletion/parser/tokenizer.cpp b/src/plugins/codecompletion/parser/tokenizer.cpp
index 2176ab8..fee843d 100644
--- a/src/plugins/codecompletion/parser/tokenizer.cpp
+++ b/src/plugins/codecompletion/parser/tokenizer.cpp
@@ -1236,6 +1236,9 @@ void Tokenizer::ReplaceMacro(wxString& str)
                 }
             }
         }
+        // if in macro expansion mode, we don't let the manually replacement rule
+        // executed again, so just returned
+        return;
     }
 
     wxStringHashMap::const_iterator it = s_Replacements.find(str);
@@ -1296,6 +1299,15 @@ void Tokenizer::ReplaceMacro(wxString& str)
             str = DoGetToken();
         }
     }
+    else if (it->second[0] == _T('@'))
+    {
+        // trigger the macro replacement, so that we can look up the keyword in TokenTree for a
+        // macro usage, note that ReplaceBufferText() below just move backword the index, but
+        // don't change the buffer
+        if(ReplaceBufferText(it->first, false))
+            str = DoGetToken();
+
+    }
     else // for other user defined rules, just do a buffer content replacement
     {
         if (it->second != str && ReplaceBufferText(it->second, false))
@@ -1686,7 +1698,8 @@ void Tokenizer::SplitArguments(wxArrayString& results)
         else if (token == _T(")"))
             --level;
 
-        if (token == _T(","))
+        // comma is a delimit only it is not wrapper by ()
+        if (token == _T(",") && level == 1)
         {
             results.Add(piece);
             piece.Clear();

You need to add a replacement rule in CC setting: _EXFUN  -> @

And the result can be shown in the screen shot below:
(http://i683.photobucket.com/albums/vv194/ollydbg_cb/2014-06-20082306_zps9b3fbc29.png)
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on June 24, 2014, 04:55:31 am
In trunk now.

Code
Revision: 9829
Author: ollydbg
Date: 2014-6-24 10:48:08
Message:
* CC: add a new CC replacement rule: XXXXX -> @, and document the ReplaceMacro() function. The new rule triggers the tokenizer to switch to macro replacement mode, thus a macro definition can be looked up in the token tree. This fixes a bug report here: http://forums.codeblocks.org/index.php/topic,19278.0.html.
-------------------------------
M(T ) : /trunk/src/plugins/codecompletion/parser/tokenizer.cpp

M(T ) : /trunk/src/plugins/codecompletion/parser/tokenizer.h

I'm also updating the wiki page by adding this rule:
http://wiki.codeblocks.org/index.php?title=Code_Completion_Design

Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Teybeo on July 30, 2014, 12:02:42 am
Does this patch enables support for glew.h functions definitions ?

Example for the OpenGL function
Code
glMultiDrawElements(GLenum mode, const GLsizei *count, GLenum type, const GLvoid **indices, GLsizei drawcount);

Code
typedef void (GLAPIENTRY * PFNGLMULTIDRAWELEMENTSPROC) (GLenum mode, const GLsizei *count, GLenum type, const GLvoid **indices, GLsizei drawcount);
...
GLEW_FUN_EXPORT PFNGLMULTIDRAWELEMENTSPROC __glewMultiDrawElements;
...
#define glMultiDrawElements GLEW_GET_FUN(__glewMultiDrawElements)


#define GLEW_FUN_EXPORT GLEWAPI
#define GLEW_GET_FUN(x) x

#ifdef GLEW_STATIC
#  define GLEWAPI extern
#else
#  ifdef GLEW_BUILD
#    define GLEWAPI extern __declspec(dllexport)
#  else
#    define GLEWAPI extern __declspec(dllimport)
#  endif
#endif


Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 04, 2014, 02:30:18 am
Does this patch enables support for glew.h functions definitions ?

Example for the OpenGL function
Code
glMultiDrawElements(GLenum mode, const GLsizei *count, GLenum type, const GLvoid **indices, GLsizei drawcount);

This is a function pointer, not a function declaration, right? EDIT, true, that this is a function pointer, but we can call it like a normal function call like glMultiDrawElements(...), see Function Pointers in C and C++ - Cprogramming.com (http://www.cprogramming.com/tutorial/function-pointers.html)

I just tested, and currently, CC only regard the glMultiDrawElements as a macro definition. What is the expect behavior do you want?

EDIT:
The correct test code could be:
Code
#ifdef GLEW_STATIC
#  define GLEWAPI extern
#else
#  ifdef GLEW_BUILD
#    define GLEWAPI extern __declspec(dllexport)
#  else
#    define GLEWAPI extern __declspec(dllimport)
#  endif
#endif

#define GLEW_FUN_EXPORT GLEWAPI
#define GLEW_GET_FUN(x) x

typedef void (GLAPIENTRY * PFNGLMULTIDRAWELEMENTSPROC) (GLenum mode, const GLsizei *count, GLenum type, const GLvoid **indices, GLsizei drawcount);

GLEW_FUN_EXPORT PFNGLMULTIDRAWELEMENTSPROC __glewMultiDrawElements;

#define glMultiDrawElements GLEW_GET_FUN(__glewMultiDrawElements)

glMultiDrawElements

Or even simpiler
Code
#define GLEW_FUN_EXPORT extern
#define GLEW_GET_FUN(x) x

typedef void ( * PFNGLMULTIDRAWELEMENTSPROC) (int a);

GLEW_FUN_EXPORT PFNGLMULTIDRAWELEMENTSPROC __glewMultiDrawElements;

#define glMultiDrawElements GLEW_GET_FUN(__glewMultiDrawElements)

glMultiDrawElements


Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 05, 2014, 05:01:42 pm
Today I was about to submit my patch for function calltips, and coincidentally found this post. :) My patch adds calltips support for some complex macro functions, and variables of function pointer typedefs. Both are required to support the glew functions.

1) Advanced macro functions: supported cases,
- Preprocosser token defined to function name, macro name or variable of typedef'd function pointer.
- Nested cases of the above (defines and macro calls are expanded).
Code
void FUNC(...);
#define DEFINE FUNC
DEFINE( // show calltip for FUNC()

#define MACRO(..) ...
#define DEFINE MACRO
DEFINE( // show calltip for MACRO()
MACRO( // same

#define WRAPPER(_x) FUNC
#define DEFINE WRAPPER(x)
DEFINE( // show calltip for FUNC()

2) Fixed the support for typedef'd function pointer (there was already some non-working code for it).
Code
typedef void (* t_pfnMyProc) (int a);
t_pfnMyProc myProc;
myProc( // show calltip
Also combinations of 1) and 2) are supported (like your glew test code).

3) Some improvement and notes for GetMacroExpendedText() which is used for above features.

I'm attaching the patch file.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 05, 2014, 05:14:31 pm
I'm sleep now, will look into it tomorrow, thanks.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 06, 2014, 08:02:14 am
I try to test this code with your patch:
Code
void my_f(int b);

void (*foo)(int a);

foo = &my_f;

foo(5);

But I don't have tip info if the mouse hover on the "foo(5)".
Find declaration of "foo(5)" gives nothing.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 06, 2014, 08:56:12 am
I try to test this code with your patch:
Code
void my_f(int b);

void (*foo)(int a);

foo = &my_f;

foo(5);

But I don't have tip info if the mouse hover on the "foo(5)".
Find declaration of "foo(5)" gives nothing.

I think normal function pointer variables parsing require another patch (I will send it next, but it requires some cleanup before). This patch supports typedef'd function pointers only (and it's a patch for function calltips, not tooltips).
Try this:
Code
void my_f(int b);

typedef void (*t_foo)(int a);
t_foo foo;

foo = my_f; // I think we should assign function directly, not &my_f
            // EDIT: Ok, in fact both my_f and &my_f are correct...

foo( // shows calltip for void t_foo(int a)
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 07, 2014, 04:38:10 pm
Hi, Huki, thanks for the explanation, when testing and reviewing your patch, I see this comment which I'm not clear:

Code
                    // NOTE: ReplaceFunctionLikeMacro will recursively expand all defines and macro calls,
                    // but will not expand macro names, which is what we want
                    smallTokenizer.ReplaceFunctionLikeMacro(tk);
                    tk = tree->at(tree->TokenExists(smallTokenizer.GetToken(), -1, tkFunction|tkMacroDef|tkVariable));
                    if (tk && smallTokenizer.PeekToken().empty()) // only if the expanded result is a single token
                        token = tk;
What does the "will not expand macro names" in comments means? as I see, ReplaceFunctionLikeMacro() did a full expansion until nothing is expanded.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 07, 2014, 06:06:51 pm
Hi, Huki, thanks for the explanation, when testing and reviewing your patch, I see this comment which I'm not clear:

Code
                    // NOTE: ReplaceFunctionLikeMacro will recursively expand all defines and macro calls,
                    // but will not expand macro names, which is what we want
                    smallTokenizer.ReplaceFunctionLikeMacro(tk);
                    tk = tree->at(tree->TokenExists(smallTokenizer.GetToken(), -1, tkFunction|tkMacroDef|tkVariable));
                    if (tk && smallTokenizer.PeekToken().empty()) // only if the expanded result is a single token
                        token = tk;
What does the "will not expand macro names" in comments means? as I see, ReplaceFunctionLikeMacro() did a full expansion until nothing is expanded.
Hi,
Consider a macro definition like this:
Code
#define MACRO(_x) ...
It must be used like this (this is a macro call):
Code
MACRO(5);

Now we have a define pointing to this macro:
Code
#define DEFINE MACRO
Note that there is no "()" after MACRO.. so this is not a valid "macro call", just the macro name is used alone. In normal code this is illegal and probably an error, but when used like this in the define, it means the preprocessor token DEFINE should be used like a macro:
Code
DEFINE(5); // same as MACRO(5);
So to qualify as a macro name, the macro definition should have arguments (or empty parentheses), but the macro usage should be without any parentheses.

In ReplaceFunctionLikeMacro() I have added this test to make sure such macro names don't get expanded (expanding them is illegal anyway).
Code
@@ -1954,16 +1962,26 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
[...]
+    // don't replace anything if the arguments are missing
+    if (!SplitArguments(actualArgs))
         return false;

BTW: valid preprocessor tokens will still be expanded (i.e., ordinary preprocessor tokens that have no parentheses in the definition itself, like #define DEFINE ...
I added that feature too in the patch:
Code
@@ -1937,6 +1938,13 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
     if (tk->m_FullType.Find(tk->m_Name) != wxNOT_FOUND)
         return false;
 
+    // if not even "()" is found [in the definition] then it's a normal preprocessor define, so handle as such
+    if (tk->m_Args.IsEmpty())
+    {
+        expandedText = tk->m_FullType;
+        return true;    // return true for ReplaceBufferText()
+    }
+

Also valid macro calls will get expanded:
Code
#define GLEW_GET_FUN(x) x

// now
#define glMultiDrawElements GLEW_GET_FUN(__glewMultiDrawElements)
// becomes
#define glMultiDrawElements __glewMultiDrawElements

// show calltip for __glewMultiDrawElements(...), which can be
// a function, macro, function ptr or typedef'd function ptr
// (currently function ptr are not parsed properly in DoParse() so
//  it's not supported for now. But typedef'd func ptrs are supported)
glMultiDrawElements(

Likewise we have function calls and function names, see some example code:
Code
// MessageBoxA is a function name
// it means we can call MessageBox() and get calltip for MessageBoxA()
#define MessageBox MessageBoxA

// MessageBoxA is a function CALL
// it means we cannot use MessageBox like a call, it's just a define.
#define MessageBox MessageBoxA(...)
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 08, 2014, 08:19:24 am
Hi,
Consider a macro definition like this:
Code
#define MACRO(_x) ...
It must be used like this (this is a macro call):
Code
MACRO(5);

Now we have a define pointing to this macro:
Code
#define DEFINE MACRO
Note that there is no "()" after MACRO.. so this is not a valid "macro call", just the macro name is used alone. In normal code this is illegal and probably an error,
Thanks for the explain.
It looks like this code above is valid, and I just create a simple test code, and it build fine under MinGW G++
Code
#define MACRO(_x) _x
#define DEFINE MACRO

int main()
{
    int x;
    DEFINE(x);
    return 0;
}


Quote
but when used like this in the define, it means the preprocessor token DEFINE should be used like a macro:
Code
DEFINE(5); // same as MACRO(5);
So to qualify as a macro name, the macro definition should have arguments (or empty parentheses), but the macro usage should be without any parentheses.
What does the last sentence means? I don't get the idea.

Quote
In ReplaceFunctionLikeMacro() I have added this test to make sure such macro names don't get expanded (expanding them is illegal anyway).
Sorry, I still not quite understand this sentence. (Maybe, dur to my poor English.....)

Quote
Code
@@ -1954,16 +1962,26 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
[...]
+    // don't replace anything if the arguments are missing
+    if (!SplitArguments(actualArgs))
         return false;
I see that the only usage of the GetMacroExpendedText() is here: (BTW: typo here? should be "expanded text"?)
Code
bool Tokenizer::ReplaceFunctionLikeMacro(const Token* tk, bool updatePeekToken)
{
    wxString macroExpandedText;
    if ( GetMacroExpendedText(tk, macroExpandedText) )
        return ReplaceBufferText(macroExpandedText, updatePeekToken);
    return false;
}
But as I see that the function ReplaceFunctionLikeMacro is called in some condition that tk->m_Args is not empty. It looks like m_Args is used to distinguish whether a macro definition is variable-like or function-like, but if I realize that even function-like macro definitions are allowed to use empty formal argument list, so the condition is not true any more.

GetMacroExpendedText did some tricks, it just put the formal arguments before actual arguments, so the buffer becomes
Code
.....  ( formal arguments )  ( actual arguments ) ....
       ^ --- m_TokenIndex

Oh, so for a function-like macro definition which has empty argument list, the m_Args is "()".  SplitArguments(actualArgs) function will still return true, but both the actualArgs  and the formalArgs are empty,  in this case, we can directly return the macro's definition string without any replacement.

BTW: it looks like ReplaceFunctionLikeMacro() only runs one level replacement, if you have several replacement rules (A -> B ->C), call this function only make a (A->B), but I think (B->C) will be happens after some GetToken() call. So, it looks like our macro expansion is not similar as the C language standard.


Quote
BTW: valid preprocessor tokens will still be expanded (i.e., ordinary preprocessor tokens that have no parentheses in the definition itself, like #define DEFINE ...
I added that feature too in the patch:
Code
@@ -1937,6 +1938,13 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
     if (tk->m_FullType.Find(tk->m_Name) != wxNOT_FOUND)
         return false;
 
+    // if not even "()" is found [in the definition] then it's a normal preprocessor define, so handle as such
+    if (tk->m_Args.IsEmpty())
+    {
+        expandedText = tk->m_FullType;
+        return true;    // return true for ReplaceBufferText()
+    }
+
Yes, this is some kind of expanding variable-like macro.

Edit: I think I still need time to review the CC code and your patches.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 08, 2014, 08:55:18 am
Some test:
Here is the test code
Code
#define AAA BBB
#define BBB CCC
#define CCC DDD
#define DDD EEE

int AAA;

Nornally, when parsed, you get a variable Token named "AAA", since CC don't check every token for macros, so AAA does not trigger macro replacement.

If you create a user replacement rule by adding AAA -> @ in the CC's setting(this just move back the m_TokenIndex, and put the Tokenizer in macro replacement mode), you will get a variable Token named "EEE", which is correct.

I see that DoGetToken() can be recurively called, thus recursive macro replacement happens when DoGetToken() is called.

EDIT:

In your patch
Code
                    //NOTE: ReplaceFunctionLikeMacro will recursively expand all defines and macro calls,
                    // but will not expand macro names, which is what we want
                    smallTokenizer.ReplaceFunctionLikeMacro(tk);
I think the comment is not correct right? Since ReplacefunctionLikeMacro just do text replacement once.

EDIT2:
If I change to this code
Code
#define AAA BBB
#define BBB CCC
#define CCC DDD
#define DDD EEE
#define EEE BBB

int AAA;
You will quickly get an infinite loop, until we run to the limit:
Code
    if (m_RepeatReplaceCount > 0)
    {
        if (m_RepeatReplaceCount >= s_MaxRepeatReplaceCount)
        {
            m_TokenIndex = m_BufferLen - m_FirstRemainingLength;
            m_PeekAvailable = false;
To solve this issue, I think
1, collecting the used macros, and don't use it again
2, expand all the macros once, not by recursive call of DoGetToken



Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 08, 2014, 10:50:27 am

EDIT:

In your patch
Code
                    //NOTE: ReplaceFunctionLikeMacro will recursively expand all defines and macro calls,
                    // but will not expand macro names, which is what we want
                    smallTokenizer.ReplaceFunctionLikeMacro(tk);

I see that when you call ReplaceFunctionLikeMacro(), then the Tokenizer goes to the macro replacement mode, so the next time you call smallTokenizer.GetToken(), macro replacement will recursively expand all.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 08, 2014, 11:30:25 am
Oh, sorry about the confusion. Regarding macro expansion: for any macro, we have 1) the macro definition, and 2) the macro usage. Each of them can be of two kinds: 1) having parentheses (...), or 2) not having parentheses. So you can see there are totally 4 cases to handle. Then the macro can either be simple, or nested (needs recursive expansion). I will try to explain the different cases one by one.

Case 1: Both macro definition and macro usage do not have (). Eg,
Code
#define FIVE 5
int a = FIVE;
We can call this a variable-like macro define / usage.

Case 2: Both macro definition and macro usage have (). Eg,
Code
#define MUL(_x) (_x * 5)
int a = MUL(1);
We can call this a function-like macro define / usage.

Case 3: Macro definition does not have () but macro usage does have it. Eg,
Code
#define MUL(_x) (_x * 5)

#define OPER MUL

// OPER(1) is an example of Case 3 macro usage
int a = OPER(1);
Here, OPER is defined as a variable-macro, but used as a function-macro. We should expand it like a variable-macro using plain ReplaceBufferText(). In my patch I have added support for this case (the test added at the beginning of GetMacroExpendedText()).

Case 4: Macro definition has the () but macro usage does not have it. Eg,
Code
#define MUL(_x) (_x * 5)

// MUL is an example of Case 4 macro usage
#define OPER MUL // this line is valid (but CC should not expand MUL)

// these two lines are valid
int a = OPER(1);
int b = MUL(1);

int c = MUL; // this line is invalid (CC should not expand MUL)

Here, MUL is defined as a function-like macro. But in the line "#define OPER MUL" it is used without parentheses. This means our CC engine should treat it like an error and leave it alone (do not expand it). We can use this behavior to stop our recursive expansion at this point, and use the final result "MUL" for the calltips.

Quote from: ollydbg
It looks like this code above is valid, and I just create a simple test code, and it build fine under MinGW G++
[...]
Yes, this is valid, but still we should not expand it (we need to stop there and use the MUL token for our calltips). Imagine if we expand it:
Code
#define OPER MUL
// becomes
#define OPER (_x * 5) // invalid because OPER cannot accept any arguments!
So generally our macro expansion code should always treat Case 4 like an error and don't expand it...

That is the reason for this test added:
Code
@@ -1954,16 +1962,26 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
[...]
+    // don't replace anything if the [actual (or usage)] arguments are missing
+    if (!SplitArguments(actualArgs))
         return false;

I hope that is clear now.

Quote from: ollydbg
I see that the only usage of the GetMacroExpendedText() is here: (BTW: typo here? should be "expanded text"?)
Yes, here is the only usage of this function. GetMacroExpendedText() should return "true" if we want ReplaceBufferText(), and "false" if we want to leave the macro alone and don't replace. But even if there is empty () in the macro definition we have to handle it in GetMacroExpendedText() before returning true.
And yes, it should be "expanded text". :)

Quote from: ollydbg
It looks like m_Args is used to distinguish whether a macro definition is variable-like or function-like, but if I realize that even function-like macro definitions are allowed to use empty formal argument list, so the condition is not true any more.
Yes, but m_Args is not empty even if there is empty parentheses. So we can safely check for m_Args->IsEmpty(). A function-macro like this: "#define MACRO() ..." will have in m_Args: "()". Only for variable-macro the m_Args is really empty.

Quote from: ollydbg
BTW: it looks like ReplaceFunctionLikeMacro() only runs one level replacement, if you have several replacement rules (A -> B ->C), call this function only make a (A->B), but I think (B->C) will be happens after some GetToken() call.
Quote from: ollydbg
I think the comment is not correct right? Since ReplacefunctionLikeMacro just do text replacement once.
Well, yes, the actual recursive expansion happens in the next GetToken() call, but I wanted to keep the comments simple and easy to read. Technically, ReplacefunctionLikeMacro() expands one level, and turns on the flag for further expansion in the next GetToken() or PeekToken().
In my patch there is:
Code
+                    smallTokenizer.ReplaceFunctionLikeMacro(tk);
+                    tk = tree->at(tree->TokenExists(smallTokenizer.GetToken(), ...
So, full recursive expansion occurs in the GetToken(), but when we hit Case 4, GetMacroExpendedText() will return false, and it won't be expanded further. We use that result for the calltips.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 08, 2014, 04:22:01 pm
...
I hope that is clear now.
Hi, Huki, thanks for the explanation, now I fully understand the idea of your patch. Especially the Case 3 and Case 4.
As a conclusion: Showing the call tip is very tricky. Sometimes, you get some text matched macro tokens, but those tokens don't have args. How can we show call tips for them? We just expand those tokens, the most important(final) step to collect the call tip information is that expansion should stop at a Token which has args.
Yeah, the args are the call tips. Thus, don't expand to much, since we may lose the args if we expand to much.  ;)
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 09, 2014, 04:28:42 pm
When testing your patch, I found another bug in CC's call tip.
Code::Blocks / Tickets / #47 Show call tip of the constructor function (https://sourceforge.net/p/codeblocks/tickets/47/)
Call tip failed with or without your patch. So, let's leave it for another fix.  ;)
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 10, 2014, 04:55:25 am
@Huki, your patch is in trunk now! Thanks for the contribution. I change some comments, fell free to comment.  ;D
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 10, 2014, 04:52:13 pm
Thanks for the commit.. and the comments. :)

When testing your patch, I found another bug in CC's call tip.
Code::Blocks / Tickets / #47 Show call tip of the constructor function (https://sourceforge.net/p/codeblocks/tickets/47/)
Call tip failed with or without your patch. So, let's leave it for another fix.  ;)
I get the bug with your test code. But curiously it works inside the CB project, see the attached screenshots: both types of constructor calltips work fine. I remember it was added recently by a patch from Alpha, I found the post where you mention it, see discussion here (http://forums.codeblocks.org/index.php/topic,13753.msg130203.html#msg130203) (at the end of the post).

EDIT: removed attachments.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 10, 2014, 05:05:32 pm
Thanks for the commit.. and the comments. :)

When testing your patch, I found another bug in CC's call tip.
Code::Blocks / Tickets / #47 Show call tip of the constructor function (https://sourceforge.net/p/codeblocks/tickets/47/)
Call tip failed with or without your patch. So, let's leave it for another fix.  ;)
I get the bug with your test code. But curiously it works inside the CB project, see the attached screenshots: both types of constructor calltips work fine. I remember it was added recently by a patch from Alpha, I found the post where you mention it, see discussion here (http://forums.codeblocks.org/index.php/topic,13753.msg130203.html#msg130203) (at the end of the post).

Oh, I just found that the below test works OK for showing the call tip, see the "public" key word.
Code
class AAA
{
public:
    AAA(int a){;}
    int m_b;
};


AAA(8);

 :D  :D
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 10, 2014, 05:31:52 pm
Oh, I just found that the below test works OK for showing the call tip, see the "public" key word.
Code
class AAA
{
public:
    AAA(int a){;}
    int m_b;
};


AAA(8);

 :D  :D
Ahah.. :P Yep, that works for me too.


BTW, now I also tested my note about semi-colon:
Quote
I think Alpha's patch was trying to get calltips for:
Code
wxStaticText static(|
Though I'm not sure if it can work, because the variable "static" in the above code won't be parsed unless semi-colon is typed, right?
See the code in DoParse(): if we peek a "(" or a ";", we parse the variable.
Code
                else if (   (peek == ParserConsts::semicolon)
                         || (   (   m_Options.useBuffer
                                 && (peek.GetChar(0) == ParserConsts::opbracket_chr) )
                             && (!m_Str.Contains(ParserConsts::dcolon)) ) )
                {
                    if (   !m_Str.IsEmpty()
                        && (    wxIsalpha(token.GetChar(0))
                            || (token.GetChar(0) == ParserConsts::underscore_chr) ) )
                    {
                        if (m_Options.handleVars)
                        {
                            Token* newToken = DoAddToken(tkVariable, token, m_Tokenizer.GetLineNumber());
                            if (newToken && !m_TemplateArgument.IsEmpty())
                                ResolveTemplateArgs(newToken);
                        }
                        else
                            SkipToOneOfChars(ParserConsts::semicolonclbrace, true, true);
                    }
                }

The current code is already checking for the opbracket, but there seem to be some problems:
Code
class AAA
{
public:
    AAA(int a){;}
    int m_b;
};

AAA bbb( // I don't get the constructor calltip

EDIT: removed attachments.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Alpha on September 10, 2014, 06:02:28 pm
Oh, I just found that the below test works OK for showing the call tip, see the "public" key word.
Sorry, should I have mentioned earlier.  I filtered out non-visible ctors to avoid the annoying wxString(int) and similar (private ctors are almost never used, to my knowledge).
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 11, 2014, 07:15:25 am
The current code is already checking for the opbracket, but there seem to be some problems:
Code
class AAA
{
public:
    AAA(int a){;}
    int m_b;
};

AAA bbb( // I don't get the constructor calltip

EDIT: removed attachments.


In my opinion, the "variable pattern" can only be recognized as: "AAA BBB;"  or "AAA BBB CCC" or "AAA BBB, CCC".
AAA, BBB, CCC are all kinds of lexeme.
But the pattern like "AAA BBB(...);" are recognized as functions.

If we see "AAA BBB(...)" we don't know it is a variable or a function. :) That's the reason of this call tip failure I think.
I will check the code now.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 11, 2014, 07:21:13 am
Oh, I just found that the below test works OK for showing the call tip, see the "public" key word.
Sorry, should I have mentioned earlier.  I filtered out non-visible ctors to avoid the annoying wxString(int) and similar (private ctors are almost never used, to my knowledge).
Yes, thanks, that should be the expected behavior too. I guess ollydbg and I were getting a little forgetful.. ;)


BTW, now I also tested my note about semi-colon:
[...]
Today I checked it and it's clear now. A code like AAA bbb(...) can be either a function declaration, or a class variable initialized with a constructor. So how do we find out which one it is? Technically the class variable won't have type info in the arguments, eg:
Code
AAA bbb(int a); // function declaration
AAA bbb(a); // variable of class AAA

But our CC plugin follows a much simpler method: if the code is found in the global namespace, "handle_functions" is true, and it is parsed like a function declaration. Inside a local block, "handle_functions" is false and it is parsed like a variable. So with the current code we can get the calltip:
Code
class AAA
{
public:
    AAA(int a){;}
    int m_b;
};

void func()
{
    AAA bbb( // shows calltip for constructor AAA::AAA()
}

It's probably enough for most purposes, so for now I'll leave it like this.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 11, 2014, 07:27:07 am
Oh, I just found that the below test works OK for showing the call tip, see the "public" key word.
Sorry, should I have mentioned earlier.  I filtered out non-visible ctors to avoid the annoying wxString(int) and similar (private ctors are almost never used, to my knowledge).
Yes, thanks, that should be the expected behavior too. I guess ollydbg and I were getting a little forgetful.. ;)
Yes, I can't remember anything about this.   ;)


Quote
BTW, now I also tested my note about semi-colon:
[...]
Today I checked it and it's clear now. A code like AAA bbb(...) can be either a function declaration, or a class variable initialized with a constructor. So how do we find out which one it is? Technically the class variable won't have type info in the arguments, eg:
Code
AAA bbb(int a); // function declaration
AAA bbb(a); // variable of class AAA

But our CC plugin follows a much simpler method: if the code is found in the global namespace, "handle_functions" is true, and it is parsed like a function declaration. Inside a local block, "handle_functions" is false and it is parsed like a variable. So with the current code we can get the calltip:
Code
class AAA
{
public:
    AAA(int a){;}
    int m_b;
};

void func()
{
    AAA bbb( // shows calltip for constructor AAA::AAA()
}

It's probably enough for most purposes, so for now I'll leave it like this.

Correct, I just review the code, and I get the same conclusion. It comes from this code:
Code
   (peek == ParserConsts::semicolon)
                         || (   (   m_Options.useBuffer
                                 && (peek.GetChar(0) == ParserConsts::opbracket_chr) )
                             && (!m_Str.Contains(ParserConsts::dcolon)) ) )

m_Options.useBuffer means we are parsing function body, and peek.GetChar(0) == ParserConsts::opbracket_chr means we find a pattern like "AAA BBB(", so this is a variable only in function bodies. Not easy to handle those issue, since they need semantic information.

EDIT
I will add some comments about those patterns in the source code.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 11, 2014, 11:32:17 am
I have looked at the patch for function pointers that I mentioned earlier:
I try to test this code with your patch:
Code
void my_f(int b);

void (*foo)(int a);

foo = &my_f;

foo(5);

But I don't have tip info if the mouse hover on the "foo(5)".
Find declaration of "foo(5)" gives nothing.

I think normal function pointer variables parsing require another patch (I will send it next, but it requires some cleanup before).

Function pointers are handled just like normal functions, but of course, they only have a declaration, not definition. So I've updated HandleFunctions() to handle this as a special case.

This patch adds support for func ptr both in global and local block. Also, all the cases involving opbracket_chr are handled in one place, in this order:
Code
void(*foo)(int a); // func ptr
void *(*foo)(int a); // func ptr with ptr return type
void foo(int a); // function decl
AAA foo(5); // var initialized with ctor (only supported for local block)

You can use your earlier test code (quoted above) to test this patch. Let me know if it's ok.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 12, 2014, 04:15:54 am
I have looked at the patch for function pointers that I mentioned earlier:
I try to test this code with your patch:
Code
void my_f(int b);

void (*foo)(int a);

foo = &my_f;

foo(5);

But I don't have tip info if the mouse hover on the "foo(5)".
Find declaration of "foo(5)" gives nothing.

I think normal function pointer variables parsing require another patch (I will send it next, but it requires some cleanup before).

Function pointers are handled just like normal functions, but of course, they only have a declaration, not definition. So I've updated HandleFunctions() to handle this as a special case.

This patch adds support for func ptr both in global and local block. Also, all the cases involving opbracket_chr are handled in one place, in this order:
Code
void(*foo)(int a); // func ptr
void *(*foo)(int a); // func ptr with ptr return type
void foo(int a); // function decl
AAA foo(5); // var initialized with ctor (only supported for local block)

You can use your earlier test code (quoted above) to test this patch. Let me know if it's ok.

Thanks, I reviewed and tested the patch, works fine, thanks! Here is my modified patch. (I just add some comments)
Code
 src/plugins/codecompletion/parser/parser.cpp       |   2 +-
 src/plugins/codecompletion/parser/parserthread.cpp | 101 +++++++++++++++++++--
 src/plugins/codecompletion/parser/parserthread.h   |   3 +-
 3 files changed, 94 insertions(+), 12 deletions(-)

diff --git a/src/plugins/codecompletion/parser/parser.cpp b/src/plugins/codecompletion/parser/parser.cpp
index feeec27..a7886bd 100644
--- a/src/plugins/codecompletion/parser/parser.cpp
+++ b/src/plugins/codecompletion/parser/parser.cpp
@@ -631,7 +631,7 @@ bool Parser::ParseBuffer(const wxString& buffer,   bool isLocal,
     opts.wantPreprocessor     = m_Options.wantPreprocessor;
     opts.parseComplexMacros   = false;
 
-    opts.handleFunctions      = false;
+    opts.handleFunctions      = true;   // enabled to support function ptr in local block
 
     opts.storeDocumentation   = m_Options.storeDocumentation;
 
diff --git a/src/plugins/codecompletion/parser/parserthread.cpp b/src/plugins/codecompletion/parser/parserthread.cpp
index d408270..5d5ab6c 100644
--- a/src/plugins/codecompletion/parser/parserthread.cpp
+++ b/src/plugins/codecompletion/parser/parserthread.cpp
@@ -1004,6 +1004,8 @@ void ParserThread::DoParse()
         }
         else if (!switchHandled)
         {
+            // since we can't recognize the pattern by token, then the token
+            // is normally an identifier style lexme, now we try to peek the next token
             wxString peek = m_Tokenizer.PeekToken();
             if (!peek.IsEmpty())
             {
@@ -1014,8 +1016,9 @@ void ParserThread::DoParse()
                     && m_EncounteredTypeNamespaces.empty()
                     && (!m_LastParent || m_LastParent->m_Name != token) ) // if func has same name as current scope (class)
                 {
+                    // pattern: AAA (...)
                     int id = m_TokenTree->TokenExists(token, -1, tkMacroDef);
-
+                    // if AAA is a macro definition, then expand this macro
                     if (id != -1)
                     {
                         HandleMacroExpansion(id, peek);
@@ -1023,17 +1026,31 @@ void ParserThread::DoParse()
                     }
                     else
                     {
+                        // see what is inside the (...)
                         wxString arg = m_Tokenizer.GetToken(); // eat args ()
+                        // try to see whether the peek pattern is (* BBB)
                         int pos = peek.find(ParserConsts::ptr);
                         if (pos != wxNOT_FOUND)
                         {
                             peek = m_Tokenizer.PeekToken();
                             if (peek.GetChar(0) == ParserConsts::opbracket_chr)
                             {
+                                // pattern: AAA (* BBB) (...)
+                                // where peek is (...) and arg is (* BBB)
                                 arg.Trim(true).RemoveLast();
                                 arg.Remove(0, pos+1);
-                                if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
-                                    HandleFunction(arg); // function
+                                arg.Trim(true).Trim(false);
+                                // NOTE: support func ptr in local block, show return type.
+                                // if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
+                                //     HandleFunction(arg); // function
+                                // AAA now becomes the last element of stacked type string
+                                // which is the return type of function ptr
+                                m_Str << token << ParserConsts::space_chr;
+                                // * BBB is now the function ptr's name
+                                HandleFunction(/*function name*/ arg,
+                                               /*isOperator*/    false,
+                                               /*isPointer*/     true);
+                                m_Str.Clear();
                             }
                         }
                         else // wxString arg = m_Tokenizer.GetToken(); // eat args ()
@@ -1042,10 +1059,50 @@ void ParserThread::DoParse()
                 }
                 else if (peek.GetChar(0) == ParserConsts::opbracket_chr && m_Options.handleFunctions)
                 {
-                    if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
+                    // NOTE: support some more cases..., such as m_Str is not empty
+                    // if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
+                    //     HandleFunction(token); // function
+                    // else
+                    //     m_Tokenizer.GetToken(); // eat args when parsing block
+
+                    // function ptr with pointer return type
+                    // eg: void *(*Alloc)(void *p, size_t size);
+                    // where, m_Str=void, token=(*Alloc), peek=(void *p, size_t size)
+                    if (   (m_LastToken == ParserConsts::ref_chr || m_LastToken == ParserConsts::ptr_chr) // (m_PointerOrRef)
+                        && (token.GetChar(0) == ParserConsts::opbracket_chr))
+                    {
+                        int pos = token.find(ParserConsts::ptr);
+                        if (pos != wxNOT_FOUND)
+                        {
+                            wxString arg = token;
+                            arg.Trim(true).RemoveLast();
+                            arg.Remove(0, pos+1);
+                            arg.Trim(true).Trim(false);
+                            HandleFunction(/*function name*/ arg,
+                                           /*isOperator*/    false,
+                                           /*isPointer*/     true);
+                        }
+                    }
+                    else if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
+                    {
+                        // pattern AAA BBB (...) in global namespace (not in local block)
+                        // so, this is mostly like a function declaration, but in-fact this
+                        // can also be a global variable initializized with ctor, but for
+                        // simplicity, we drop the later case
                         HandleFunction(token); // function
+                    }
                     else
+                    {
+                        // local variables initialized with ctor
+                        if (!m_Str.IsEmpty() && m_Options.handleVars)
+                        {
+                            Token* newToken = DoAddToken(tkVariable, token, m_Tokenizer.GetLineNumber());
+                            if (newToken && !m_TemplateArgument.IsEmpty())
+                                ResolveTemplateArgs(newToken);
+                        }
                         m_Tokenizer.GetToken(); // eat args when parsing block
+                    }
+
                     m_Str.Clear();
                 }
                 else if (   (peek  == ParserConsts::colon)
@@ -1113,15 +1170,15 @@ void ParserThread::DoParse()
                         m_EncounteredNamespaces.push(token);
                     m_Tokenizer.GetToken(); // eat ::
                 }
-                else if (   (peek == ParserConsts::semicolon)
-                         || (   (   m_Options.useBuffer
-                                 && (peek.GetChar(0) == ParserConsts::opbracket_chr) )
-                             && (!m_Str.Contains(ParserConsts::dcolon)) ) )
+                // NOTE: opbracket_chr already handled above
+                else if (peek==ParserConsts::semicolon)
                 {
                     if (   !m_Str.IsEmpty()
                         && (    wxIsalpha(token.GetChar(0))
                             || (token.GetChar(0) == ParserConsts::underscore_chr) ) )
                     {
+                        // pattern: m_Str AAA;
+                        // where AAA is the variable name, m_Str contains type string
                         if (m_Options.handleVars)
                         {
                             Token* newToken = DoAddToken(tkVariable, token, m_Tokenizer.GetLineNumber());
@@ -1597,7 +1654,7 @@ void ParserThread::HandleDefines()
     wxString para; // function-like macro's args
     if (!readToEOL.IsEmpty())
     {
-        // a '(' char follow the macro name (without space between them) is regard as a
+        // a '(' char follow the macro name (without space between them) is regard as a
         // function like macro definition
         if (readToEOL[0] == ParserConsts::opbracket_chr) // function-like macro definition
         {
@@ -2091,7 +2148,7 @@ void ParserThread::HandleClass(EClassType ct)
     m_Tokenizer.SetState(oldState);
 }
 
-void ParserThread::HandleFunction(const wxString& name, bool isOperator)
+void ParserThread::HandleFunction(const wxString& name, bool isOperator, bool isPointer)
 {
     TRACE(_T("HandleFunction() : Adding function '")+name+_T("': m_Str='")+m_Str+_T("'"));
     int lineNr = m_Tokenizer.GetLineNumber();
@@ -2099,6 +2156,30 @@ void ParserThread::HandleFunction(const wxString& name, bool isOperator)
     wxString peek = m_Tokenizer.PeekToken();
     TRACE(_T("HandleFunction() : name='")+name+_T("', args='")+args+_T("', peek='")+peek+_T("'"));
 
+    // special case for function pointers
+    if (isPointer)
+    {
+        // pattern: m_Str AAA (*BBB) (...);
+        if (peek == ParserConsts::semicolon)
+        {
+            TRACE(_T("HandleFunction() : Add token name='")+name+_T("', args='")+args+_T("', return type='") + m_Str+ _T("'"));
+            Token* newToken =  DoAddToken(tkFunction, name, lineNr, 0, 0, args);
+            if (newToken)
+            {
+                newToken->m_IsConst = false;
+                newToken->m_TemplateArgument = m_TemplateArgument;
+                if (!m_TemplateArgument.IsEmpty() && newToken->m_TemplateMap.empty())
+                    ResolveTemplateArgs(newToken);
+            }
+            else
+            {
+                TRACE(_T("HandleFunction() : Unable to create/add new token: ") + name);
+            }
+            m_TemplateArgument.Clear();
+        }
+        return;
+    }
+
     if (!m_Str.StartsWith(ParserConsts::kw_friend))
     {
         int lineStart = 0;
diff --git a/src/plugins/codecompletion/parser/parserthread.h b/src/plugins/codecompletion/parser/parserthread.h
index d81b752..7421177 100644
--- a/src/plugins/codecompletion/parser/parserthread.h
+++ b/src/plugins/codecompletion/parser/parserthread.h
@@ -200,8 +200,9 @@ protected:
     /** handle function declaration or definition
       * @param name function name
       * @param isOperator if true, means it is an operator overload function
+      * @param isPointer if true, means it is a function pointer
       */
-    void HandleFunction(const wxString& name, bool isOperator = false);
+    void HandleFunction(const wxString& name, bool isOperator = false, bool isPointer = false);
 
     /** parse for loop arguments:
       * for(int X; ... ; ...)


If you are OK,  I'm going to commit.

BTW: by the way, maybe, the two condition:

Code
                if (   (peek.GetChar(0) == ParserConsts::opbracket_chr)
                    && m_Options.handleFunctions
                    && m_Str.IsEmpty()
                    && m_EncounteredNamespaces.empty()
                    && m_EncounteredTypeNamespaces.empty()
                    && (!m_LastParent || m_LastParent->m_Name != token) )
                {
                       Handle pattern AAA(...)
                       -> maybe, it is a macro usage
                       -> maybe, it is a function ptr
                }

Code
                else if (peek.GetChar(0) == ParserConsts::opbracket_chr && m_Options.handleFunctions)
                {
                       Handle pattern CCC::DDD  EEE::AAA(...)
                       or                  CCC::DDD* EEE::AAA(...)
                       -> maybe, it is a function declaration
                       -> maybe, it is a function ptr
                       -> maybe, it is a local variable initialization with ctor
                }
Those two conditions can be merged or some refactored, but I'm not quite sure. E.g. extract the handling macro usage, and merge handling of function decl or function ptr in one condition. This can be a new commit.  :D

BTW2: coding style: when adding comment, I think "// NOTE" is better, there is a space after "//".


Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 12, 2014, 03:01:55 pm
Ok about the patch, so feel free to commit. :)

Quote
BTW2: coding style: when adding comment, I think "// NOTE" is better, there is a space after "//".
Ok about that too.

BTW: by the way, maybe, the two condition:
[...]
Those two conditions can be merged or some refactored, but I'm not quite sure. E.g. extract the handling macro usage, and merge handling of function decl or function ptr in one condition. This can be a new commit.  :D
I agree we can separate the macro handling and merge the function handling.
We can think about supporting more cases for macro handling too. We currently handle function-like macros, and only when m_Str is empty, eg:
Code
some code;
MACRO(); // expand this
more code;
Sometimes the macro is not the first token in the line, so m_Str is non-empty, just like the bug reported by the original poster in this thread:
Code
FILE * _EXFUN(fopen, (const char *__restrict _name, const char *__restrict _type));
Those can be handled.. and also variable-like macros..
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 12, 2014, 04:38:16 pm
Ok about the patch, so feel free to commit. :)
Done. rev 9916, thanks!


Quote
BTW: by the way, maybe, the two condition:
[...]
Those two conditions can be merged or some refactored, but I'm not quite sure. E.g. extract the handling macro usage, and merge handling of function decl or function ptr in one condition. This can be a new commit.  :D
I agree we can separate the macro handling and merge the function handling.
We can think about supporting more cases for macro handling too. We currently handle function-like macros, and only when m_Str is empty, eg:
Code
some code;
MACRO(); // expand this
more code;
Sometimes the macro is not the first token in the line, so m_Str is non-empty, just like the bug reported by the original poster in this thread:
Code
FILE * _EXFUN(fopen, (const char *__restrict _name, const char *__restrict _type));
Those can be handled.. and also variable-like macros..
Good catch, though it will increase the parsing time, but I think it is worthy. This can remove my own hack on the user defined macro replacement rule.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: White-Tiger on September 14, 2014, 12:23:03 pm
Yay! for the function pointer patch xD It finally works^^ (ok didn't test it, but I'm using them in different projects and they didn't work most of the time..)
It's weird though.. I've thought Code::Blocks was able to handle them as they sometimes seemed to work.. just didn't most of the time :P
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 14, 2014, 02:28:32 pm
Yay! for the function pointer patch xD It finally works^^ (ok didn't test it, but I'm using them in different projects and they didn't work most of the time..)
It's weird though.. I've thought Code::Blocks was able to handle them as they sometimes seemed to work.. just didn't most of the time :P
Does the trunk fix your but report here Re: The 06 July 2014 build (9844) is out. (http://forums.codeblocks.org/index.php/topic,19428.msg133098.html#msg133098)?
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: White-Tiger on September 14, 2014, 04:30:30 pm
yes an no.. the reported case works, same for the real world code I based it on... (at least it looks like that, didn't miss any function)
But another case I've used to work with, does not work.

I'll post the almost non-stripped code just in case there's another "bug" (had to strip it a bit because of 20k character limit for posts^^)
Code: c++
#include <stdint.h>
typedef uint64_t uint64;
typedef unsigned short anyID;
#define TS3_VECTOR anyID
enum LogLevel{a,b,c,};
#define PluginMessageTarget LogLevel
#define PluginItemType LogLevel
#define PluginGuiProfile LogLevel
#define PluginConnectTab LogLevel


struct TS3Functions {
unsigned int (*getClientLibVersion)(char** result);
unsigned int (*getClientLibVersionNumber)(uint64* result);
unsigned int (*spawnNewServerConnectionHandler)(int port, uint64* result);
unsigned int (*destroyServerConnectionHandler)(uint64 serverConnectionHandlerID);

/* Error handling */
unsigned int (*getErrorMessage)(unsigned int errorCode, char** error);

/* Memory management */
unsigned int (*freeMemory)(void* pointer);

/* Logging */
unsigned int (*logMessage)(const char* logMessage, enum LogLevel severity, const char* channel, uint64 logID);

/* Sound */
unsigned int (*getPlaybackDeviceList)(const char* modeID, char**** result);
unsigned int (*getPlaybackModeList)(char*** result);
unsigned int (*getCaptureDeviceList)(const char* modeID, char**** result);
unsigned int (*getCaptureModeList)(char*** result);
unsigned int (*getDefaultPlaybackDevice)(const char* modeID, char*** result);
unsigned int (*getDefaultPlayBackMode)(char** result);
unsigned int (*getDefaultCaptureDevice)(const char* modeID, char*** result);
unsigned int (*getDefaultCaptureMode)(char** result);
unsigned int (*openPlaybackDevice)(uint64 serverConnectionHandlerID, const char* modeID, const char* playbackDevice);
unsigned int (*openCaptureDevice)(uint64 serverConnectionHandlerID, const char* modeID, const char* captureDevice);
unsigned int (*getCurrentPlaybackDeviceName)(uint64 serverConnectionHandlerID, char** result, int* isDefault);
unsigned int (*getCurrentPlayBackMode)(uint64 serverConnectionHandlerID, char** result);
unsigned int (*getCurrentCaptureDeviceName)(uint64 serverConnectionHandlerID, char** result, int* isDefault);
unsigned int (*getCurrentCaptureMode)(uint64 serverConnectionHandlerID, char** result);
unsigned int (*initiateGracefulPlaybackShutdown)(uint64 serverConnectionHandlerID);
unsigned int (*closePlaybackDevice)(uint64 serverConnectionHandlerID);
unsigned int (*closeCaptureDevice)(uint64 serverConnectionHandlerID);
unsigned int (*activateCaptureDevice)(uint64 serverConnectionHandlerID);
unsigned int (*playWaveFileHandle)(uint64 serverConnectionHandlerID, const char* path, int loop, uint64* waveHandle);
unsigned int (*pauseWaveFileHandle)(uint64 serverConnectionHandlerID, uint64 waveHandle, int pause);
unsigned int (*closeWaveFileHandle)(uint64 serverConnectionHandlerID, uint64 waveHandle);
unsigned int (*playWaveFile)(uint64 serverConnectionHandlerID, const char* path);
unsigned int (*registerCustomDevice)(const char* deviceID, const char* deviceDisplayName, int capFrequency, int capChannels, int playFrequency, int playChannels);
unsigned int (*unregisterCustomDevice)(const char* deviceID);
unsigned int (*processCustomCaptureData)(const char* deviceName, const short* buffer, int samples);
unsigned int (*acquireCustomPlaybackData)(const char* deviceName, short* buffer, int samples);

/* Preprocessor */
unsigned int (*getPreProcessorInfoValueFloat)(uint64 serverConnectionHandlerID, const char* ident, float* result);
unsigned int (*getPreProcessorConfigValue)(uint64 serverConnectionHandlerID, const char* ident, char** result);
unsigned int (*setPreProcessorConfigValue)(uint64 serverConnectionHandlerID, const char* ident, const char* value);

/* Encoder */
unsigned int (*getEncodeConfigValue)(uint64 serverConnectionHandlerID, const char* ident, char** result);

/* Playback */
unsigned int (*getPlaybackConfigValueAsFloat)(uint64 serverConnectionHandlerID, const char* ident, float* result);
unsigned int (*setPlaybackConfigValue)(uint64 serverConnectionHandlerID, const char* ident, const char* value);
unsigned int (*setClientVolumeModifier)(uint64 serverConnectionHandlerID, anyID clientID, float value);

/* Recording */
unsigned int (*startVoiceRecording)(uint64 serverConnectionHandlerID);
unsigned int (*stopVoiceRecording)(uint64 serverConnectionHandlerID);

/* 3d sound positioning */
unsigned int (*systemset3DListenerAttributes) (uint64 serverConnectionHandlerID, const TS3_VECTOR* position, const TS3_VECTOR* forward, const TS3_VECTOR* up);
unsigned int (*set3DWaveAttributes) (uint64 serverConnectionHandlerID, uint64 waveHandle, const TS3_VECTOR* position);
unsigned int (*systemset3DSettings) (uint64 serverConnectionHandlerID, float distanceFactor, float rolloffScale);
unsigned int (*channelset3DAttributes) (uint64 serverConnectionHandlerID, anyID clientID, const TS3_VECTOR* position);

/* Interaction with the server */
unsigned int (*startConnection)(uint64 serverConnectionHandlerID, const char* identity, const char* ip, unsigned int port, const char* nickname,
                               const char** defaultChannelArray, const char* defaultChannelPassword, const char* serverPassword);
unsigned int (*stopConnection)(uint64 serverConnectionHandlerID, const char* quitMessage);
unsigned int (*requestClientMove)(uint64 serverConnectionHandlerID, anyID clientID, uint64 newChannelID, const char* password, const char* returnCode);
unsigned int (*requestClientVariables)(uint64 serverConnectionHandlerID, anyID clientID, const char* returnCode);
unsigned int (*requestClientKickFromChannel)(uint64 serverConnectionHandlerID, anyID clientID, const char* kickReason, const char* returnCode);
unsigned int (*requestClientKickFromServer)(uint64 serverConnectionHandlerID, anyID clientID, const char* kickReason, const char* returnCode);
unsigned int (*requestChannelDelete)(uint64 serverConnectionHandlerID, uint64 channelID, int force, const char* returnCode);
unsigned int (*requestChannelMove)(uint64 serverConnectionHandlerID, uint64 channelID, uint64 newChannelParentID, uint64 newChannelOrder, const char* returnCode);
unsigned int (*requestSendPrivateTextMsg)(uint64 serverConnectionHandlerID, const char* message, anyID targetClientID, const char* returnCode);
unsigned int (*requestSendChannelTextMsg)(uint64 serverConnectionHandlerID, const char* message, uint64 targetChannelID, const char* returnCode);
unsigned int (*requestSendServerTextMsg)(uint64 serverConnectionHandlerID, const char* message, const char* returnCode);
unsigned int (*requestConnectionInfo)(uint64 serverConnectionHandlerID, anyID clientID, const char* returnCode);
unsigned int (*requestClientSetWhisperList)(uint64 serverConnectionHandlerID, anyID clientID, const uint64* targetChannelIDArray, const anyID* targetClientIDArray, const char* returnCode);
unsigned int (*requestChannelSubscribe)(uint64 serverConnectionHandlerID, const uint64* channelIDArray, const char* returnCode);
unsigned int (*requestChannelSubscribeAll)(uint64 serverConnectionHandlerID, const char* returnCode);
unsigned int (*requestChannelUnsubscribe)(uint64 serverConnectionHandlerID, const uint64* channelIDArray, const char* returnCode);
unsigned int (*requestChannelUnsubscribeAll)(uint64 serverConnectionHandlerID, const char* returnCode);
unsigned int (*requestChannelDescription)(uint64 serverConnectionHandlerID, uint64 channelID, const char* returnCode);
unsigned int (*requestMuteClients)(uint64 serverConnectionHandlerID, const anyID* clientIDArray, const char* returnCode);
unsigned int (*requestUnmuteClients)(uint64 serverConnectionHandlerID, const anyID* clientIDArray, const char* returnCode);
unsigned int (*requestClientPoke)(uint64 serverConnectionHandlerID, anyID clientID, const char* message, const char* returnCode);
unsigned int (*requestClientIDs)(uint64 serverConnectionHandlerID, const char* clientUniqueIdentifier, const char* returnCode);
unsigned int (*clientChatClosed)(uint64 serverConnectionHandlerID, const char* clientUniqueIdentifier, anyID clientID, const char* returnCode);
unsigned int (*clientChatComposing)(uint64 serverConnectionHandlerID, anyID clientID, const char* returnCode);
unsigned int (*requestServerTemporaryPasswordAdd)(uint64 serverConnectionHandlerID, const char* password, const char* description, uint64 duration, uint64 targetChannelID, const char* targetChannelPW, const char* returnCode);
unsigned int (*requestServerTemporaryPasswordDel)(uint64 serverConnectionHandlerID, const char* password, const char* returnCode);
unsigned int (*requestServerTemporaryPasswordList)(uint64 serverConnectionHandlerID, const char* returnCode);

/* Access clientlib information */

/* Query own client ID */
unsigned int (*getClientID)(uint64 serverConnectionHandlerID, anyID* result);

/* Client info */
unsigned int (*getClientSelfVariableAsInt)(uint64 serverConnectionHandlerID, size_t flag, int* result);
unsigned int (*getClientSelfVariableAsString)(uint64 serverConnectionHandlerID, size_t flag, char** result);
unsigned int (*setClientSelfVariableAsInt)(uint64 serverConnectionHandlerID, size_t flag, int value);
unsigned int (*setClientSelfVariableAsString)(uint64 serverConnectionHandlerID, size_t flag, const char* value);
unsigned int (*flushClientSelfUpdates)(uint64 serverConnectionHandlerID, const char* returnCode);
unsigned int (*getClientVariableAsInt)(uint64 serverConnectionHandlerID, anyID clientID, size_t flag, int* result);
unsigned int (*getClientVariableAsUInt64)(uint64 serverConnectionHandlerID, anyID clientID, size_t flag, uint64* result);
unsigned int (*getClientVariableAsString)(uint64 serverConnectionHandlerID, anyID clientID, size_t flag, char** result);
unsigned int (*getClientList)(uint64 serverConnectionHandlerID, anyID** result);
unsigned int (*getChannelOfClient)(uint64 serverConnectionHandlerID, anyID clientID, uint64* result);

/* Channel info */
unsigned int (*getChannelVariableAsInt)(uint64 serverConnectionHandlerID, uint64 channelID, size_t flag, int* result);
unsigned int (*getChannelVariableAsUInt64)(uint64 serverConnectionHandlerID, uint64 channelID, size_t flag, uint64* result);
unsigned int (*getChannelVariableAsString)(uint64 serverConnectionHandlerID, uint64 channelID, size_t flag, char** result);
unsigned int (*getChannelIDFromChannelNames)(uint64 serverConnectionHandlerID, char** channelNameArray, uint64* result);
unsigned int (*setChannelVariableAsInt)(uint64 serverConnectionHandlerID, uint64 channelID, size_t flag, int value);
unsigned int (*setChannelVariableAsUInt64)(uint64 serverConnectionHandlerID, uint64 channelID, size_t flag, uint64 value);
unsigned int (*setChannelVariableAsString)(uint64 serverConnectionHandlerID, uint64 channelID, size_t flag, const char* value);
unsigned int (*flushChannelUpdates)(uint64 serverConnectionHandlerID, uint64 channelID, const char* returnCode);
unsigned int (*flushChannelCreation)(uint64 serverConnectionHandlerID, uint64 channelParentID, const char* returnCode);
unsigned int (*getChannelList)(uint64 serverConnectionHandlerID, uint64** result);
unsigned int (*getChannelClientList)(uint64 serverConnectionHandlerID, uint64 channelID,  anyID** result);
unsigned int (*getParentChannelOfChannel)(uint64 serverConnectionHandlerID, uint64 channelID, uint64* result);

/* Server info */
unsigned int (*getServerConnectionHandlerList)(uint64** result);
unsigned int (*getServerVariableAsInt)(uint64 serverConnectionHandlerID, size_t flag, int* result);
unsigned int (*getServerVariableAsUInt64)(uint64 serverConnectionHandlerID, size_t flag, uint64* result);
unsigned int (*getServerVariableAsString)(uint64 serverConnectionHandlerID, size_t flag, char** result);
unsigned int (*requestServerVariables)(uint64 serverConnectionHandlerID);

/* Connection info */
unsigned int (*getConnectionStatus)(uint64 serverConnectionHandlerID, int* result);
unsigned int (*getConnectionVariableAsUInt64)(uint64 serverConnectionHandlerID, anyID clientID, size_t flag, uint64* result);
unsigned int (*getConnectionVariableAsDouble)(uint64 serverConnectionHandlerID, anyID clientID, size_t flag, double* result);
unsigned int (*getConnectionVariableAsString)(uint64 serverConnectionHandlerID, anyID clientID, size_t flag, char** result);
unsigned int (*cleanUpConnectionInfo)(uint64 serverConnectionHandlerID, anyID clientID);

/* Client related */
unsigned int (*requestClientDBIDfromUID)(uint64 serverConnectionHandlerID, const char* clientUniqueIdentifier, const char* returnCode);
unsigned int (*requestClientNamefromUID)(uint64 serverConnectionHandlerID, const char* clientUniqueIdentifier, const char* returnCode);
unsigned int (*requestClientNamefromDBID)(uint64 serverConnectionHandlerID, uint64 clientDatabaseID, const char* returnCode);
unsigned int (*requestClientEditDescription)(uint64 serverConnectionHandlerID, anyID clientID, const char* clientDescription, const char* returnCode);
unsigned int (*requestClientSetIsTalker)(uint64 serverConnectionHandlerID, anyID clientID, int isTalker, const char* returnCode);
unsigned int (*requestIsTalker)(uint64 serverConnectionHandlerID, int isTalkerRequest, const char* isTalkerRequestMessage, const char* returnCode);

/* Plugin related */
unsigned int (*requestSendClientQueryCommand)(uint64 serverConnectionHandlerID, const char* command, const char* returnCode);

/* Filetransfer */
unsigned int (*getTransferFileName)(anyID transferID, char** result);
unsigned int (*getTransferFilePath)(anyID transferID, char** result);
unsigned int (*getTransferFileSize)(anyID transferID, uint64* result);
unsigned int (*getTransferFileSizeDone)(anyID transferID, uint64* result);
unsigned int (*isTransferSender)(anyID transferID, int* result);  /* 1 == upload, 0 == download */
unsigned int (*getTransferStatus)(anyID transferID, int* result);
unsigned int (*getCurrentTransferSpeed)(anyID transferID, float* result);
unsigned int (*getAverageTransferSpeed)(anyID transferID, float* result);
unsigned int (*getTransferRunTime)(anyID transferID, uint64* result);
unsigned int (*sendFile)(uint64 serverConnectionHandlerID, uint64 channelID, const char* channelPW, const char* file, int overwrite, int resume, const char* sourceDirectory, anyID* result, const char* returnCode);
unsigned int (*requestFile)(uint64 serverConnectionHandlerID, uint64 channelID, const char* channelPW, const char* file, int overwrite, int resume, const char* destinationDirectory, anyID* result, const char* returnCode);
unsigned int (*haltTransfer)(uint64 serverConnectionHandlerID, anyID transferID, int deleteUnfinishedFile, const char* returnCode);
unsigned int (*requestFileList)(uint64 serverConnectionHandlerID, uint64 channelID, const char* channelPW, const char* path, const char* returnCode);
unsigned int (*requestFileInfo)(uint64 serverConnectionHandlerID, uint64 channelID, const char* channelPW, const char* file, const char* returnCode);
unsigned int (*requestDeleteFile)(uint64 serverConnectionHandlerID, uint64 channelID, const char* channelPW, const char** file, const char* returnCode);
unsigned int (*requestCreateDirectory)(uint64 serverConnectionHandlerID, uint64 channelID, const char* channelPW, const char* directoryPath, const char* returnCode);
unsigned int (*requestRenameFile)(uint64 serverConnectionHandlerID, uint64 fromChannelID, const char* channelPW, uint64 toChannelID, const char* toChannelPW, const char* oldFile, const char* newFile, const char* returnCode);

/** [...] **/

/* Client functions */
void         (*getAppPath)(char* path, size_t maxLen);
void         (*getResourcesPath)(char* path, size_t maxLen);
void         (*getConfigPath)(char* path, size_t maxLen);
void         (*getPluginPath)(char* path, size_t maxLen);
uint64       (*getCurrentServerConnectionHandlerID)();
void         (*printMessage)(uint64 serverConnectionHandlerID, const char* message, enum PluginMessageTarget messageTarget);
void         (*printMessageToCurrentTab)(const char* message);
void         (*urlsToBB)(const char* text, char* result, size_t maxLen);
void         (*sendPluginCommand)(uint64 serverConnectionHandlerID, const char* pluginID, const char* command, int targetMode, const anyID* targetIDs, const char* returnCode);
void         (*getDirectories)(const char* path, char* result, size_t maxLen);
unsigned int (*getServerConnectInfo)(uint64 scHandlerID, char* host, unsigned short* port, char* password, size_t maxLen);
unsigned int (*getChannelConnectInfo)(uint64 scHandlerID, uint64 channelID, char* path, char* password, size_t maxLen);
void         (*createReturnCode)(const char* pluginID, char* returnCode, size_t maxLen);
unsigned int (*requestInfoUpdate)(uint64 scHandlerID, enum PluginItemType itemType, uint64 itemID);
uint64       (*getServerVersion)(uint64 scHandlerID);
unsigned int (*isWhispering)(uint64 scHandlerID, anyID clientID, int* result);
unsigned int (*isReceivingWhisper)(uint64 scHandlerID, anyID clientID, int* result);
unsigned int (*getAvatar)(uint64 scHandlerID, anyID clientID, char* result, size_t maxLen);
void         (*setPluginMenuEnabled)(const char* pluginID, int menuID, int enabled);
void         (*showHotkeySetup)();
void         (*requestHotkeyInputDialog)(const char* pluginID, const char* keyword, int isDown, void* qParentWindow);
unsigned int (*getHotkeyFromKeyword)(const char* pluginID, const char** keywords, char** hotkeys, size_t arrayLen, size_t hotkeyBufSize);
unsigned int (*getClientDisplayName)(uint64 scHandlerID, anyID clientID, char* result, size_t maxLen);
unsigned int (*getBookmarkList)(struct PluginBookmarkList** list);
unsigned int (*getProfileList)(enum PluginGuiProfile profile, int* defaultProfileIdx, char*** result);
unsigned int (*guiConnect)(enum PluginConnectTab connectTab, const char* serverLabel, const char* serverAddress, const char* serverPassword, const char* nickname, const char* channel, const char* channelPassword, const char* captureProfile, const char* playbackProfile, const char* hotkeyProfile, const char* soundProfile, const char* userIdentity, const char* oneTimeKey, const char* phoneticName, uint64* scHandlerID);
unsigned int (*guiConnectBookmark)(enum PluginConnectTab connectTab, const char* bookmarkuuid, uint64* scHandlerID);
unsigned int (*createBookmark)(const char* bookmarkuuid, const char* serverLabel, const char* serverAddress, const char* serverPassword, const char* nickname, const char* channel, const char* channelPassword, const char* captureProfile, const char* playbackProfile, const char* hotkeyProfile, const char* soundProfile, const char* uniqueUserId, const char* oneTimeKey, const char* phoneticName);
unsigned int (*getPermissionIDByName)(uint64 serverConnectionHandlerID, const char* permissionName, unsigned int* result);
unsigned int (*getClientNeededPermission)(uint64 serverConnectionHandlerID, const char* permissionName, int* result);
};
TS3Functions.

Basically the problem already appears at the first function... at "getClientLibVersion"... it's detected... but only as
public int(* getClientLibVersion): unsigned
So there's a problem with unsigned int as a return because it only picks up the first identifier...

It also corrupts the code-completion popup... sometimes you'll get a different declaration to the right then, the selected function on the left...
Looks like this: (http://i.imgur.com/qMJ9Ey6m.png) (http://i.imgur.com/qMJ9Ey6.png)
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 15, 2014, 07:00:24 am
Quote
BTW: by the way, maybe, the two condition:
[...]
Those two conditions can be merged or some refactored, but I'm not quite sure. E.g. extract the handling macro usage, and merge handling of function decl or function ptr in one condition. This can be a new commit.  :D
I agree we can separate the macro handling and merge the function handling.
We can think about supporting more cases for macro handling too. We currently handle function-like macros, and only when m_Str is empty, eg:
Code
some code;
MACRO(); // expand this
more code;
Sometimes the macro is not the first token in the line, so m_Str is non-empty, just like the bug reported by the original poster in this thread:
Code
FILE * _EXFUN(fopen, (const char *__restrict _name, const char *__restrict _type));
Those can be handled.. and also variable-like macros..
Good catch, though it will increase the parsing time, but I think it is worthy. This can remove my own hack on the user defined macro replacement rule.

Maybe, a better method is to try expand every identifier like token if possible, I remembered you have a patch named "cc_parser_general.patch", but when I looked at that patch in my PC, I see that patch contains too many things. (a lot of them is already in trunk)

To enable this feature, we can just do this patch:
Code
 src/plugins/codecompletion/parser/tokenizer.cpp | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/plugins/codecompletion/parser/tokenizer.cpp b/src/plugins/codecompletion/parser/tokenizer.cpp
index f47e8dd..31f0546 100644
--- a/src/plugins/codecompletion/parser/tokenizer.cpp
+++ b/src/plugins/codecompletion/parser/tokenizer.cpp
@@ -1246,8 +1246,8 @@ wxString Tokenizer::DoGetToken()
 void Tokenizer::ReplaceMacro(wxString& str)
 {
     // this indicates we are already in macro replacement mode
-    if (m_RepeatReplaceCount > 0)
-    {
+//    if (m_RepeatReplaceCount > 0)
+//    {
         const int id = m_TokenTree->TokenExists(str, -1, tkMacroDef);
         if (id != -1)
         {
@@ -1270,8 +1270,9 @@ void Tokenizer::ReplaceMacro(wxString& str)
         // if in macro expansion mode, we don't want to let the user replacement rule executed
         // again, so just returned
         return;
-    }
+//    }
 
+#if 0
     wxStringHashMap::const_iterator it = s_Replacements.find(str);
     if (it == s_Replacements.end())
         return;
@@ -1344,6 +1345,7 @@ void Tokenizer::ReplaceMacro(wxString& str)
         if (it->second != str && ReplaceBufferText(it->second, false))
             str = DoGetToken();
     }
+#endif
 }
 
 bool Tokenizer::CalcConditionExpression()

Thus, we totally remove all the user defined replacement rules.
I just test the patch, the parsing time is a bit longer, but not too much.  ;D
Another smart method is that we can only check the macro usage on the identifier like token which has all capital characters or underscore.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 15, 2014, 07:34:39 am
yes an no.. the reported case works, same for the real world code I based it on... (at least it looks like that, didn't miss any function)
But another case I've used to work with, does not work.

I'll post the almost non-stripped code just in case there's another "bug" (had to strip it a bit because of 20k character limit for posts^^)
...

Basically the problem already appears at the first function... at "getClientLibVersion"... it's detected... but only as
public int(* getClientLibVersion): unsigned
So there's a problem with unsigned int as a return because it only picks up the first identifier...

Thanks for the sample code.
OK, I can strip the test code (can be run in cctest project)
Code

struct TS3Functions {
unsigned int (*getClientLibVersion)(char** result);
unsigned int (*getClientLibVersionNumber)(uint64* result);
};

TS3Functions a;


//a.    //getClientLibVersion

The result is:
Code
000001. --------------M-a-i-n--L-o-g--------------


000002. -----------I-n-t-e-r-i-m--L-o-g-----------
000003. InitTokenizer() : m_Filename='F:\cb_sf_git\trunk\src\plugins\codecompletion\testing\ccc_function_ptr.cpp', m_FileSize=255.
000004. Init() : m_Filename='F:\cb_sf_git\trunk\src\plugins\codecompletion\testing\ccc_function_ptr.cpp'
000005. F:\cb_sf_git\trunk\src\plugins\codecompletion\testing\ccc_function_ptr.cpp
000006. Parse() : Parsing 'F:\cb_sf_git\trunk\src\plugins\codecompletion\testing\ccc_function_ptr.cpp'
000007. SkipComment() : Start from line = 1
000008. DoParse() : Loop:m_Str='', token='struct'
000009. HandleClass() : Found class 'TS3Functions', next='{'
000010. GetRealTypeIfTokenIsMacro() : tokenIsMacro=no -> tokenName='TS3Functions'.
000011. DoAddToken() : Created token='TS3Functions', file_idx=1, line=5, ticket=256
000012. GetTokenBaseType() : Searching within m_Str=''
000013. GetTokenBaseType() : Compensated m_Str=''
000014. GetTokenBaseType() : Returning ''
000015. DoAddToken() : Prepending ''
000016. DoAddToken() : Added/updated token 'TS3Functions' (0), kind 'class', type '', actual ''. Parent is  (-1)
000017. DoParse() : Loop:m_Str='', token='unsigned'
000018. DoParse() : Loop:m_Str='unsigned ', token='int'
000019. ReadParentheses(): (* getClientLibVersion), line=6
000020. HandleFunction() : Adding function 'int': m_Str='unsigned '
000021. ReadParentheses(): (char** result), line=6
000022. HandleFunction() : name='int', args='(* getClientLibVersion)', peek='(char** result)'
000023. HandleFunction() : !(Ctor/Dtor) 'int', m_Str='unsigned ', localParent='<none>'
000024. HandleFunction() : Adding function 'int', ': m_Str='unsigned ', enc_ns='nil'.
000025. HandleFunction() : Possible macro '(char** result)' in function 'int' (file name='F:\cb_sf_git\trunk\src\plugins\codecompletion\testing\ccc_function_ptr.cpp', line numer 6).
000026. HandleFunction() : Add token name='int', args='(* getClientLibVersion)', return type='unsigned '
000027. GetBaseArgs() : args='(* getClientLibVersion)'.
000028. GetBaseArgs() : baseArgs='(*)'.
000029. DoAddToken() : Created token='int', file_idx=1, line=6, ticket=257
000030. GetTokenBaseType() : Searching within m_Str='unsigned'
000031. GetTokenBaseType() : Compensated m_Str='unsigned'
000032. GetTokenBaseType() : Found 'unsigned'
000033. DoAddToken() : Prepending ''
000034. DoAddToken() : Added/updated token 'int' (1), kind 'function', type 'unsigned', actual 'unsigned'. Parent is TS3Functions (0)
000035. DoParse() : Loop:m_Str='', token='(char** result)'
000036. DoParse() : Loop:m_Str='', token=';'
000037. DoParse() : Loop:m_Str='', token='unsigned'
000038. DoParse() : Loop:m_Str='unsigned ', token='int'
000039. ReadParentheses(): (* getClientLibVersionNumber), line=7
000040. HandleFunction() : Adding function 'int': m_Str='unsigned '
000041. ReadParentheses(): (uint64* result), line=7
000042. HandleFunction() : name='int', args='(* getClientLibVersionNumber)', peek='(uint64* result)'
000043. HandleFunction() : !(Ctor/Dtor) 'int', m_Str='unsigned ', localParent='<none>'
000044. HandleFunction() : Adding function 'int', ': m_Str='unsigned ', enc_ns='nil'.
000045. HandleFunction() : Possible macro '(uint64* result)' in function 'int' (file name='F:\cb_sf_git\trunk\src\plugins\codecompletion\testing\ccc_function_ptr.cpp', line numer 7).
000046. HandleFunction() : Add token name='int', args='(* getClientLibVersionNumber)', return type='unsigned '
000047. GetBaseArgs() : args='(* getClientLibVersionNumber)'.
000048. GetBaseArgs() : baseArgs='(*)'.
000049. DoAddToken() : Found token (parent).
000050. GetTokenBaseType() : Searching within m_Str='unsigned'
000051. GetTokenBaseType() : Compensated m_Str='unsigned'
000052. GetTokenBaseType() : Found 'unsigned'
000053. DoAddToken() : Prepending ''
000054. DoAddToken() : Added/updated token 'int' (1), kind 'function', type 'unsigned', actual 'unsigned'. Parent is TS3Functions (0)
000055. DoParse() : Loop:m_Str='', token='(uint64* result)'
000056. DoParse() : Loop:m_Str='', token=';'
000057. DoParse() : Loop:m_Str='', token='}'
000058. DoParse() : Loop:m_Str='', token='TS3Functions'
000059. DoParse() : Loop:m_Str='TS3Functions ', token='a'
000060. DoAddToken() : Created token='a', file_idx=1, line=10, ticket=258
000061. GetTokenBaseType() : Searching within m_Str='TS3Functions'
000062. GetTokenBaseType() : Compensated m_Str='TS3Functions'
000063. GetTokenBaseType() : Found 'TS3Functions'
000064. DoAddToken() : Prepending ''
000065. DoAddToken() : Added/updated token 'a' (2), kind 'variable', type 'TS3Functions', actual 'TS3Functions'. Parent is  (-1)
000066. DoParse() : Loop:m_Str='TS3Functions', token=';'
000067. SkipComment() : Start from line = 13
000068. SkipComment() : Need to call SkipToInlineCommentEnd() here at line = 13
000069. bool Tokenizer::SkipToInlineCommentEnd() : line=13, CurrentChar='a', PreviousChar='/', NextChar='.'
000070. SkipToInlineCommentEnd(): (END) We are now at line 13, CurrentChar='
********************************************************
  Testing in file: F:\cb_sf_git\trunk\src\plugins\codecompletion\testing\ccc_function_ptr.cpp
********************************************************
NativeParserBase::BreakUpComponents()
FindCCTokenStart() : Starting at 0 "a."
GetNextCCToken() : at 0 (a): res=
GetNextCCToken() : Done nest: at 1 (.): res=a
GetNextCCToken() : Return at 1 (.): res=a
GetCCToken() : FindCCTokenStart returned 1 "a."
GetCCToken() : GetNextCCToken returned 1 "a"
GetCCToken() : Left ""
NativeParserBase::GenerateResultSet_2()
Find 1 valid text matched tokens from the tree.
NativeParserBase::BreakUpComponents()
FindCCTokenStart() : Starting at 0 "TS3Functions"
GetNextCCToken() : at 0 (T): res=
GetNextCCToken() : Done nest: at 12 (
GetNextCCToken() : Return at 12 (
GetCCToken() : FindCCTokenStart returned 12 "TS3Functions"
GetCCToken() : GetNextCCToken returned 12 "TS3Functions"
GetCCToken() : Left ""
NativeParserBase::GenerateResultSet_2()
Find 1 valid text matched tokens from the tree.
NativeParserBase::GenerateResultSet_2()
*FAIL: a.  getClientLibVersion
--------------------------------------------------------
Total 1 tests, 0 PASS, 1 FAIL
--------------------------------------------------------

Look: Adding function 'int': m_Str='unsigned ', the reason is we have such source code, the function name is "int".  ;D
Code
                    else if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
                    {
                        // pattern AAA BBB (...) in global namespace (not in local block)
                        // so, this is mostly like a function declaration, but in-fact this
                        // can also be a global variable initializized with ctor, but for
                        // simplicity, we drop the later case
                        HandleFunction(token); // function
                    }

and
Code
unsigned int (*getClientLibVersion)(char** result);
Satisfy "AAA BBB (...)" pattern. Maybe, we need to check whether there is a parenthesis after parenthesis.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 15, 2014, 08:29:24 am
Candidate patch to fix your problem
Code
8ee75a7f4ee217a48380bcee26ad83d674cfd489
 src/plugins/codecompletion/parser/parserthread.cpp | 23 +++++++++++++++++++++-
 src/plugins/codecompletion/parser/parserthread.h   |  2 +-
 2 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/src/plugins/codecompletion/parser/parserthread.cpp b/src/plugins/codecompletion/parser/parserthread.cpp
index 9cb2690..96ba685 100644
--- a/src/plugins/codecompletion/parser/parserthread.cpp
+++ b/src/plugins/codecompletion/parser/parserthread.cpp
@@ -2148,18 +2148,39 @@ void ParserThread::HandleClass(EClassType ct)
     m_Tokenizer.SetState(oldState);
 }
 
-void ParserThread::HandleFunction(const wxString& name, bool isOperator, bool isPointer)
+void ParserThread::HandleFunction(wxString name, bool isOperator, bool isPointer)
 {
     TRACE(_T("HandleFunction() : Adding function '")+name+_T("': m_Str='")+m_Str+_T("'"));
     int lineNr = m_Tokenizer.GetLineNumber();
     wxString args = m_Tokenizer.GetToken();
     wxString peek = m_Tokenizer.PeekToken();
+    if (peek[0] == ParserConsts::opbracket_chr)
+    {
+        // pattern unsigned int (*getClientLibVersion)(char** result);
+        // currently, m_Str = unsigned, function name = int
+        // args = (*getClientLibVersion), peek = (char** result)
+        // this may be a function pointer declaration
+        m_Str<<name;
+        name = args;    // this is in-fact the function name "getClientLibVersion"
+        int pos = name.find(ParserConsts::ptr);
+        if (pos != wxNOT_FOUND)
+        {
+            args = peek; // peek becomes the function args
+            name.Trim(true).RemoveLast();
+            name.Remove(0, pos+1);
+            name.Trim(true).Trim(false);
+            isPointer = true;
+            m_Tokenizer.GetToken(); //consume the peek
+            peek = m_Tokenizer.PeekToken(); // update the peek value
+        }
+    }
     TRACE(_T("HandleFunction() : name='")+name+_T("', args='")+args+_T("', peek='")+peek+_T("'"));
 
     // special case for function pointers
     if (isPointer)
     {
         // pattern: m_Str AAA (*BBB) (...);
+        // name = *BBB, args = (...)
         if (peek == ParserConsts::semicolon)
         {
             TRACE(_T("HandleFunction() : Add token name='")+name+_T("', args='")+args+_T("', return type='") + m_Str+ _T("'"));
diff --git a/src/plugins/codecompletion/parser/parserthread.h b/src/plugins/codecompletion/parser/parserthread.h
index 7421177..3e26189 100644
--- a/src/plugins/codecompletion/parser/parserthread.h
+++ b/src/plugins/codecompletion/parser/parserthread.h
@@ -202,7 +202,7 @@ protected:
       * @param isOperator if true, means it is an operator overload function
       * @param isPointer if true, means it is a function pointer
       */
-    void HandleFunction(const wxString& name, bool isOperator = false, bool isPointer = false);
+    void HandleFunction(wxString name, bool isOperator = false, bool isPointer = false);
 
     /** parse for loop arguments:
       * for(int X; ... ; ...)


But I have some code suggestion issue.

Code
struct TS3Functions {
unsigned int (*getClientLibVersion)(char** result);
unsigned int (*getClientLibVersionNumber)(uint64* result);
};

TS3Functions a;
a.
Code suggest list shows Ok, but if I continue entering "get", then code suggesting list disappears, not sure why........(maybe some other local patches ;))

EDIT: update the patch, so no such issue. :)
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Teybeo on September 16, 2014, 11:05:20 pm
I remembered this thread when I saw the 9916 changelog about macros and functions stuff.
So I just download and tested the 9916 build but the post-OpenGL 1.x functions are still not detected as functions but as macros, so the calltip don't show the function parameters. OpenGL 1.x function dates back 20 years ago, we are now at OpenGL 4.5 and there are now TONS of functions which we can't see the parameters types and names.

Example:
(http://i.imgur.com/IcTzF1o.png)

Basically all the detected functions are the 20 years old and sometimes deprecated ones, and the macros are the modern ones.

I've read some of you guys posts but I didn't understand a lot. Do you think it can be supported ?
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 17, 2014, 12:10:32 am
yes an no.. the reported case works, same for the real world code I based it on... (at least it looks like that, didn't miss any function)
But another case I've used to work with, does not work.

I'll post the almost non-stripped code just in case there's another "bug" (had to strip it a bit because of 20k character limit for posts^^)
...

Basically the problem already appears at the first function... at "getClientLibVersion"... it's detected... but only as
public int(* getClientLibVersion): unsigned
So there's a problem with unsigned int as a return because it only picks up the first identifier...

Candidate patch to fix your problem
[...]
Hi, yes, we need to handle return types with more than one token, like "unsigned int". Thanks for the patch, I'll review it and the macros handling part tomorrow.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 17, 2014, 12:18:27 am
I remembered this thread when I saw the 9916 changelog about macros and functions stuff.
So I just download and tested the 9916 build but the post-OpenGL 1.x functions are still not detected as functions but as macros, so the calltip don't show the function parameters. OpenGL 1.x function dates back 20 years ago, we are now at OpenGL 4.5 and there are now TONS of functions which we can't see the parameters types and names.
The support added in build 9916 is for function calltips. Just select the function from the code completion list and start typing the opening bracket, you should see the calltip showing the arguments and return type. See the screenshot below:
(http://i.imgur.com/d6pKg3Q.png)

However the code completion list itself will only show it as a macro definition as you can see.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Teybeo on September 17, 2014, 02:52:54 pm
Oops yeah, I confirm, it works like in your screenshot, good job :) !

(Minor UI detail, it would be nice if the function calltip was showing the true name of the function and not PFNGLDRAWARRAYSINSTANCEDPROC)
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 18, 2014, 01:16:09 pm
Oops yeah, I confirm, it works like in your screenshot, good job :) !

(Minor UI detail, it would be nice if the function calltip was showing the true name of the function and not PFNGLDRAWARRAYSINSTANCEDPROC)
Yes, I think it would be better to show the variable name rather than the typedef name. See here:
(http://i.imgur.com/UL1bCGl.png)

I'll submit the patch soon...
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 18, 2014, 02:15:14 pm
yes an no.. the reported case works, same for the real world code I based it on... (at least it looks like that, didn't miss any function)
But another case I've used to work with, does not work.

I'll post the almost non-stripped code just in case there's another "bug" (had to strip it a bit because of 20k character limit for posts^^)
...

Basically the problem already appears at the first function... at "getClientLibVersion"... it's detected... but only as
public int(* getClientLibVersion): unsigned
So there's a problem with unsigned int as a return because it only picks up the first identifier...

Candidate patch to fix your problem
[...]
Hi, yes, we need to handle return types with more than one token, like "unsigned int". Thanks for the patch, I'll review it and the macros handling part tomorrow.
I think your patch will fix the problem, but maybe it's better to have all function pointers checking in one place (in DoParse()), then we just send the result to HandleFunction(). The problem is that we have a pattern like: AAA BBB (*name) (arg), where m_Str = AAA, token = BBB, peek = (*name), and we can't know if this is a function declaration or function ptr without reading the next token after peek.
But I think we can use another trick: strip the '(' in peek, and see if the next char is '*'. If it is, then it should be a function pointer. See this code:
Code
// pattern unsigned int (*getClientLibVersion)(char** result);
// currently, m_Str = unsigned, token = int, peek = (*getClientLibVersion)
// this may be a function pointer declaration, we can guess without
// reading the next token, if "peek" has a ptr char and only 1 argument
// in it.

// see what is inside the (...)
// try to see whether the peek pattern is (* BBB)
wxString arg = peek;
arg.Remove(0,1); // remove '('
if (arg.GetChar(0) == ParserConsts::ptr)
{
    arg.RemoveLast();
    arg.Remove(0,1).Trim(false); // remove '*'
    m_Str << token;
    token = m_Tokenizer.GetToken(); //consume the peek
    // BBB is now the function ptr's name
    HandleFunction(/*function name*/ arg,
                   /*isOperator*/    false,
                   /*isPointer*/     true);
}
else if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
    // function declaration
else
    // local variable initialized with ctor

Btw, some notes about the function pointer handling:
1) I saw comments like: // *BBB is now the function ptr's name
In fact we strip the '*', so the function name is just BBB.

2) About the spaces trimming: I checked the ReadParantheses() function (used in DoGetToken()), and I see it guarantees that there is no space immediately after the '(' or before the ')'. So the only place we might have to trim spaces is after removing the '*' (eg, in (* BBB) there is a space before BBB).
So I think all other Trim() calls can be safely removed, like this:
Old code:
Code
arg.Trim(true).RemoveLast();
arg.Remove(0, pos+1);
arg.Trim(true).Trim(false);
New code:
Code
arg.RemoveLast();
arg.Remove(0, pos+1).Trim(false);

I quickly tested it with some spaces and here is the result.. :)
(http://i.imgur.com/J5cXFwf.png)

Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 18, 2014, 02:28:31 pm
Quote
BTW: by the way, maybe, the two condition:
[...]
Those two conditions can be merged or some refactored, but I'm not quite sure. E.g. extract the handling macro usage, and merge handling of function decl or function ptr in one condition. This can be a new commit.  :D
I agree we can separate the macro handling and merge the function handling.
We can think about supporting more cases for macro handling too. We currently handle function-like macros, and only when m_Str is empty,
I have finally gotten around to doing it. See the result below.. :)
I'm pasting the entire code for opbracket_chr and macro defines, in DoParse(). Also, now all macros are expanded in DoParse(), but only if we reached "if (!switchHandled)".

Code
[...]

        else if (!switchHandled)
        {
            // since we can't recognize the pattern by token, then the token
            // is normally an identifier style lexme, now we try to peek the next token
            wxString peek = m_Tokenizer.PeekToken();
            if (!peek.IsEmpty())
            {
                // pattern: AAA or AAA (...)
                int id = m_TokenTree->TokenExists(token, -1, tkMacroDef);
                // if AAA is a macro definition, then expand this macro
                if (id != -1)
                {
                    HandleMacroExpansion(id, peek);
                }
                // any function like pattern
                else if (   (peek.GetChar(0) == ParserConsts::opbracket_chr)
                         && m_Options.handleFunctions )
                {
                    if (   m_Str.IsEmpty()
                        && m_EncounteredNamespaces.empty()
                        && m_EncounteredTypeNamespaces.empty()
                        && (!m_LastParent || m_LastParent->m_Name != token) ) // if func has same name as current scope (class)
                    {
                        // see what is inside the (...)
                        wxString arg = m_Tokenizer.GetToken(); // eat args ()
                        // try to see whether the peek pattern is (* BBB)
                        int pos = peek.find(ParserConsts::ptr);
                        if (pos != wxNOT_FOUND)
                        {
                            peek = m_Tokenizer.PeekToken();
                            if (peek.GetChar(0) == ParserConsts::opbracket_chr)
                            {
                                // pattern: AAA (* BBB) (...)
                                // where peek is (...) and arg is (* BBB)
                                arg.RemoveLast();
                                arg.Remove(0, pos+1).Trim(false);
                                // NOTE: support func ptr in local block, show return type.
                                // if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
                                //     HandleFunction(arg); // function
                                // AAA now becomes the last element of stacked type string
                                // which is the return type of function ptr
                                m_Str << token << ParserConsts::space_chr;
                                // BBB is now the function ptr's name
                                HandleFunction(/*function name*/ arg,
                                               /*isOperator*/    false,
                                               /*isPointer*/     true);
                                m_Str.Clear();
                            }
                        }
                        else // wxString arg = m_Tokenizer.GetToken(); // eat args ()
                            m_Str = token + arg;
                    }
                    // NOTE: support some more cases..., such as m_Str is not empty
                    // if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
                    //     HandleFunction(token); // function
                    // else
                    //     m_Tokenizer.GetToken(); // eat args when parsing block

                    // function ptr with pointer return type
                    // eg: void *(*Alloc)(void *p, size_t size);
                    // where, m_Str=void, token=(*Alloc), peek=(void *p, size_t size)
                    else if (   (m_LastToken == ParserConsts::ptr_chr) //(m_PointerOrRef)
                             && (token.GetChar(0) == ParserConsts::opbracket_chr) )
                    {
                        int pos = token.find(ParserConsts::ptr);
                        if (pos != wxNOT_FOUND)
                        {
                            wxString arg = token;
                            arg.RemoveLast();
                            arg.Remove(0, pos+1).Trim(false);
                            HandleFunction(/*function name*/ arg,
                                           /*isOperator*/    false,
                                           /*isPointer*/     true);
                            m_Str.Clear();
                        }
                    }
                    else
                    {
                        // pattern unsigned int (*getClientLibVersion)(char** result);
                        // currently, m_Str = unsigned, token = int, peek = (*getClientLibVersion)
                        // this may be a function pointer declaration, we can guess without
                        // reading the next token, if "peek" has a ptr char and only 1 argument
                        // in it.

                        // see what is inside the (...)
                        // try to see whether the peek pattern is (* BBB)
                        wxString arg = peek;
                        arg.Remove(0,1); // remove '('
                        if (arg.GetChar(0) == ParserConsts::ptr)
                        {
                            arg.RemoveLast();
                            arg.Remove(0,1).Trim(false); // remove '*'
                            m_Str << token;
                            token = m_Tokenizer.GetToken(); //consume the peek
                            // BBB is now the function ptr's name
                            HandleFunction(/*function name*/ arg,
                                           /*isOperator*/    false,
                                           /*isPointer*/     true);
                        }
                        else if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
                        {
                            // pattern AAA BBB (...) in global namespace (not in local block)
                            // so, this is mostly like a function declaration, but in-fact this
                            // can also be a global variable initializized with ctor, but for
                            // simplicity, we drop the later case
                            HandleFunction(token); // function
                        }
                        else
                        {
                            // local variables initialized with ctor
                            if (!m_Str.IsEmpty() && m_Options.handleVars)
                            {
                                Token* newToken = DoAddToken(tkVariable, token, m_Tokenizer.GetLineNumber());
                                if (newToken && !m_TemplateArgument.IsEmpty())
                                    ResolveTemplateArgs(newToken);
                            }
                            m_Tokenizer.GetToken(); // eat args when parsing block
                        }
                        m_Str.Clear();
                    }
                }
                
                else if [...]

EDIT: and here is the patch:

Code
From d3f194380a8cdc0eeae06708ece5cf06dd61dd32 Mon Sep 17 00:00:00 2001
From: huki <gk7huki@gmail.com>
Date: Thu, 18 Sep 2014 18:26:08 +0530
Subject: CC: merge function handling and update macro handling

---
 src/plugins/codecompletion/parser/parserthread.cpp | 104 ++++++++++++---------
 1 file changed, 61 insertions(+), 43 deletions(-)

diff --git a/src/plugins/codecompletion/parser/parserthread.cpp b/src/plugins/codecompletion/parser/parserthread.cpp
index e1f258d..fc3c409 100644
--- a/src/plugins/codecompletion/parser/parserthread.cpp
+++ b/src/plugins/codecompletion/parser/parserthread.cpp
@@ -1009,22 +1009,21 @@ void ParserThread::DoParse()
             wxString peek = m_Tokenizer.PeekToken();
             if (!peek.IsEmpty())
             {
-                if (   (peek.GetChar(0) == ParserConsts::opbracket_chr)
-                    && m_Options.handleFunctions
-                    && m_Str.IsEmpty()
-                    && m_EncounteredNamespaces.empty()
-                    && m_EncounteredTypeNamespaces.empty()
-                    && (!m_LastParent || m_LastParent->m_Name != token) ) // if func has same name as current scope (class)
+                // pattern: AAA or AAA (...)
+                int id = m_TokenTree->TokenExists(token, -1, tkMacroDef);
+                // if AAA is a macro definition, then expand this macro
+                if (id != -1)
                 {
-                    // pattern: AAA (...)
-                    int id = m_TokenTree->TokenExists(token, -1, tkMacroDef);
-                    // if AAA is a macro definition, then expand this macro
-                    if (id != -1)
-                    {
-                        HandleMacroExpansion(id, peek);
-                        m_Str.Clear();
-                    }
-                    else
+                    HandleMacroExpansion(id, peek);
+                }
+                // any function like pattern
+                else if (   (peek.GetChar(0) == ParserConsts::opbracket_chr)
+                         && m_Options.handleFunctions )
+                {
+                    if (   m_Str.IsEmpty()
+                        && m_EncounteredNamespaces.empty()
+                        && m_EncounteredTypeNamespaces.empty()
+                        && (!m_LastParent || m_LastParent->m_Name != token) ) // if func has same name as current scope (class)
                     {
                         // see what is inside the (...)
                         wxString arg = m_Tokenizer.GetToken(); // eat args ()
@@ -1037,16 +1036,15 @@ void ParserThread::DoParse()
                             {
                                 // pattern: AAA (* BBB) (...)
                                 // where peek is (...) and arg is (* BBB)
-                                arg.Trim(true).RemoveLast();
-                                arg.Remove(0, pos+1);
-                                arg.Trim(true).Trim(false);
+                                arg.RemoveLast();
+                                arg.Remove(0, pos+1).Trim(false);
                                 // NOTE: support func ptr in local block, show return type.
                                 // if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
                                 //     HandleFunction(arg); // function
                                 // AAA now becomes the last element of stacked type string
                                 // which is the return type of function ptr
                                 m_Str << token << ParserConsts::space_chr;
-                                // * BBB is now the function ptr's name
+                                // BBB is now the function ptr's name
                                 HandleFunction(/*function name*/ arg,
                                                /*isOperator*/    false,
                                                /*isPointer*/     true);
@@ -1056,9 +1054,6 @@ void ParserThread::DoParse()
                         else // wxString arg = m_Tokenizer.GetToken(); // eat args ()
                             m_Str = token + arg;
                     }
-                }
-                else if (peek.GetChar(0) == ParserConsts::opbracket_chr && m_Options.handleFunctions)
-                {
                     // NOTE: support some more cases..., such as m_Str is not empty
                     // if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
                     //     HandleFunction(token); // function
@@ -1068,42 +1063,65 @@ void ParserThread::DoParse()
                     // function ptr with pointer return type
                     // eg: void *(*Alloc)(void *p, size_t size);
                     // where, m_Str=void, token=(*Alloc), peek=(void *p, size_t size)
-                    if (   (m_LastToken == ParserConsts::ref_chr || m_LastToken == ParserConsts::ptr_chr) // (m_PointerOrRef)
-                        && (token.GetChar(0) == ParserConsts::opbracket_chr))
+                    else if (   (m_LastToken == ParserConsts::ptr_chr) //(m_PointerOrRef)
+                             && (token.GetChar(0) == ParserConsts::opbracket_chr) )
                     {
                         int pos = token.find(ParserConsts::ptr);
                         if (pos != wxNOT_FOUND)
                         {
                             wxString arg = token;
-                            arg.Trim(true).RemoveLast();
-                            arg.Remove(0, pos+1);
-                            arg.Trim(true).Trim(false);
+                            arg.RemoveLast();
+                            arg.Remove(0, pos+1).Trim(false);
                             HandleFunction(/*function name*/ arg,
                                            /*isOperator*/    false,
                                            /*isPointer*/     true);
+                            m_Str.Clear();
                         }
                     }
-                    else if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
-                    {
-                        // pattern AAA BBB (...) in global namespace (not in local block)
-                        // so, this is mostly like a function declaration, but in-fact this
-                        // can also be a global variable initializized with ctor, but for
-                        // simplicity, we drop the later case
-                        HandleFunction(token); // function
-                    }
                     else
                     {
-                        // local variables initialized with ctor
-                        if (!m_Str.IsEmpty() && m_Options.handleVars)
+                        // pattern unsigned int (*getClientLibVersion)(char** result);
+                        // currently, m_Str = unsigned, token = int, peek = (*getClientLibVersion)
+                        // this may be a function pointer declaration, we can guess without
+                        // reading the next token, if "peek" has a ptr char and only 1 argument
+                        // in it.
+
+                        // see what is inside the (...)
+                        // try to see whether the peek pattern is (* BBB)
+                        wxString arg = peek;
+                        arg.Remove(0,1); // remove '('
+                        if (arg.GetChar(0) == ParserConsts::ptr)
                         {
-                            Token* newToken = DoAddToken(tkVariable, token, m_Tokenizer.GetLineNumber());
-                            if (newToken && !m_TemplateArgument.IsEmpty())
-                                ResolveTemplateArgs(newToken);
+                            arg.RemoveLast();
+                            arg.Remove(0,1).Trim(false); // remove '*'
+                            m_Str << token;
+                            token = m_Tokenizer.GetToken(); //consume the peek
+                            // BBB is now the function ptr's name
+                            HandleFunction(/*function name*/ arg,
+                                           /*isOperator*/    false,
+                                           /*isPointer*/     true);
+                        }
+                        else if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
+                        {
+                            // pattern AAA BBB (...) in global namespace (not in local block)
+                            // so, this is mostly like a function declaration, but in-fact this
+                            // can also be a global variable initializized with ctor, but for
+                            // simplicity, we drop the later case
+                            HandleFunction(token); // function
                         }
-                        m_Tokenizer.GetToken(); // eat args when parsing block
+                        else
+                        {
+                            // local variables initialized with ctor
+                            if (!m_Str.IsEmpty() && m_Options.handleVars)
+                            {
+                                Token* newToken = DoAddToken(tkVariable, token, m_Tokenizer.GetLineNumber());
+                                if (newToken && !m_TemplateArgument.IsEmpty())
+                                    ResolveTemplateArgs(newToken);
+                            }
+                            m_Tokenizer.GetToken(); // eat args when parsing block
+                        }
+                        m_Str.Clear();
                     }
-
-                    m_Str.Clear();
                 }
                 else if (   (peek  == ParserConsts::colon)
                          && (token != ParserConsts::kw_private)
--
1.9.4.msysgit.0

Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 18, 2014, 03:01:53 pm
Maybe, a better method is to try expand every identifier like token if possible, I remembered you have a patch named "cc_parser_general.patch", but when I looked at that patch in my PC, I see that patch contains too many things. (a lot of them is already in trunk)

To enable this feature, we can just do this patch:
Code
 src/plugins/codecompletion/parser/tokenizer.cpp | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/plugins/codecompletion/parser/tokenizer.cpp b/src/plugins/codecompletion/parser/tokenizer.cpp
index f47e8dd..31f0546 100644
--- a/src/plugins/codecompletion/parser/tokenizer.cpp
+++ b/src/plugins/codecompletion/parser/tokenizer.cpp
@@ -1246,8 +1246,8 @@ wxString Tokenizer::DoGetToken()
 void Tokenizer::ReplaceMacro(wxString& str)
 {
     // this indicates we are already in macro replacement mode
-    if (m_RepeatReplaceCount > 0)
-    {
+//    if (m_RepeatReplaceCount > 0)
+//    {
         const int id = m_TokenTree->TokenExists(str, -1, tkMacroDef);
         if (id != -1)
         {
@@ -1270,8 +1270,9 @@ void Tokenizer::ReplaceMacro(wxString& str)
         // if in macro expansion mode, we don't want to let the user replacement rule executed
         // again, so just returned
         return;
-    }
+//    }
 
+#if 0
     wxStringHashMap::const_iterator it = s_Replacements.find(str);
     if (it == s_Replacements.end())
         return;
@@ -1344,6 +1345,7 @@ void Tokenizer::ReplaceMacro(wxString& str)
         if (it->second != str && ReplaceBufferText(it->second, false))
             str = DoGetToken();
     }
+#endif
 }
 
 bool Tokenizer::CalcConditionExpression()

Thus, we totally remove all the user defined replacement rules.
I just test the patch, the parsing time is a bit longer, but not too much.  ;D
Sure, I think we can try it, but better to keep it for another commit. ;)

Quote
Another smart method is that we can only check the macro usage on the identifier like token which has all capital characters or underscore.
Maybe it won't be required unless the parsing speed is too bad, but we will see..
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 21, 2014, 09:32:48 am
I think your patch will fix the problem, but maybe it's better to have all function pointers checking in one place (in DoParse()), then we just send the result to HandleFunction(). The problem is that we have a pattern like: AAA BBB (*name) (arg), where m_Str = AAA, token = BBB, peek = (*name), and we can't know if this is a function declaration or function ptr without reading the next token after peek.
But I think we can use another trick: strip the '(' in peek, and see if the next char is '*'. If it is, then it should be a function pointer. See this code:
Code
// pattern unsigned int (*getClientLibVersion)(char** result);
// currently, m_Str = unsigned, token = int, peek = (*getClientLibVersion)
// this may be a function pointer declaration, we can guess without
// reading the next token, if "peek" has a ptr char and only 1 argument
// in it.

// see what is inside the (...)
// try to see whether the peek pattern is (* BBB)
wxString arg = peek;
arg.Remove(0,1); // remove '('
if (arg.GetChar(0) == ParserConsts::ptr)
{
    arg.RemoveLast();
    arg.Remove(0,1).Trim(false); // remove '*'
    m_Str << token;
    token = m_Tokenizer.GetToken(); //consume the peek
    // BBB is now the function ptr's name
    HandleFunction(/*function name*/ arg,
                   /*isOperator*/    false,
                   /*isPointer*/     true);
}
else if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
    // function declaration
else
    // local variable initialized with ctor
Correct, we should recognize the function patter in DoParse(), not HandleFunction.

Quote
Btw, some notes about the function pointer handling:
1) I saw comments like: // *BBB is now the function ptr's name
In fact we strip the '*', so the function name is just BBB.
Correct, thanks.


Quote
2) About the spaces trimming: I checked the ReadParantheses() function (used in DoGetToken()), and I see it guarantees that there is no space immediately after the '(' or before the ')'. So the only place we might have to trim spaces is after removing the '*' (eg, in (* BBB) there is a space before BBB).
So I think all other Trim() calls can be safely removed, like this:
Old code:
Code
arg.Trim(true).RemoveLast();
arg.Remove(0, pos+1);
arg.Trim(true).Trim(false);
New code:
Code
arg.RemoveLast();
arg.Remove(0, pos+1).Trim(false);
...
You are right, thanks.

Quote
BTW: by the way, maybe, the two condition:
[...]
Those two conditions can be merged or some refactored, but I'm not quite sure. E.g. extract the handling macro usage, and merge handling of function decl or function ptr in one condition. This can be a new commit.  :D
I agree we can separate the macro handling and merge the function handling.
We can think about supporting more cases for macro handling too. We currently handle function-like macros, and only when m_Str is empty,
I have finally gotten around to doing it. See the result below.. :)
...
EDIT: and here is the patch:
...
I'm testing your patch now. It looks like macro expansion is enabled on every identifier like tokens.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on September 21, 2014, 07:44:47 pm
I think your patch will fix the problem, but maybe it's better to have all function pointers checking in one place (in DoParse()), then we just send the result to HandleFunction(). The problem is that we have a pattern like: AAA BBB (*name) (arg), where m_Str = AAA, token = BBB, peek = (*name), and we can't know if this is a function declaration or function ptr without reading the next token after peek.
But I think we can use another trick: strip the '(' in peek, and see if the next char is '*'. If it is, then it should be a function pointer. See this code:
Code
// pattern unsigned int (*getClientLibVersion)(char** result);
// currently, m_Str = unsigned, token = int, peek = (*getClientLibVersion)
// this may be a function pointer declaration, we can guess without
// reading the next token, if "peek" has a ptr char and only 1 argument
// in it.

// see what is inside the (...)
// try to see whether the peek pattern is (* BBB)
wxString arg = peek;
arg.Remove(0,1); // remove '('
if (arg.GetChar(0) == ParserConsts::ptr)
{
    arg.RemoveLast();
    arg.Remove(0,1).Trim(false); // remove '*'
    m_Str << token;
    token = m_Tokenizer.GetToken(); //consume the peek
    // BBB is now the function ptr's name
    HandleFunction(/*function name*/ arg,
                   /*isOperator*/    false,
                   /*isPointer*/     true);
}
else if (!m_Options.useBuffer || m_Options.bufferSkipBlocks)
    // function declaration
else
    // local variable initialized with ctor
Correct, we should recognize the function patter in DoParse(), not HandleFunction.

On second look I noticed we can do a little optimization. This code:
Code
wxString arg = peek;
arg.Remove(0,1); // remove '('
if (arg.GetChar(0) == ParserConsts::ptr)
{
    arg.Remove(0,1).Trim(false); // remove '*'
    [...]
}
can be changed to:
Code
if (peek.GetChar(1) == ParserConsts::ptr)
{
    wxString arg = peek;
    arg.Remove(0,2).Trim(false); // remove "(*"
    [...]
}

So the entire code for this case:
Code
// see what is inside the (...)
// try to see whether the peek pattern is (* BBB)
if (peek.GetChar(1) == ParserConsts::ptr)
{
    wxString arg = peek;
    arg.RemoveLast(); // remove ")"
    arg.Remove(0,2).Trim(false); // remove "(* "
    m_Str << token;
    token = m_Tokenizer.GetToken(); //consume the peek
    // BBB is now the function ptr's name
    HandleFunction(/*function name*/ arg,
                   /*isOperator*/    false,
                   /*isPointer*/     true);
}
else if [...]
That way we don't have to create a temporary variable 'arg' every time when we see a function declaration-like pattern, but only when we know for sure it can be a function pointer.


Quote
I'm testing your patch now. It looks like macro expansion is enabled on every identifier like tokens.
Yes, when the (!switchHandled) case is reached in DoParse(), it now expands all macros, including variable-like ones.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on September 22, 2014, 04:04:38 am
Quote
I'm testing your patch now. It looks like macro expansion is enabled on every identifier like tokens.
Yes, when the (!switchHandled) case is reached in DoParse(), it now expands all macros, including variable-like ones.
When I first look at this function, I see that it only expand function-like macro usage.

Code
void ParserThread::HandleMacroExpansion(int id, const wxString &peek)
{
    Token* tk = m_TokenTree->at(id);
    if (tk)
    {
        TRACE(_T("HandleMacroExpansion() : Adding token '%s' (peek='%s')"), tk->m_Name.wx_str(), peek.wx_str());
        DoAddToken(tkMacroUse, tk->m_Name, m_Tokenizer.GetLineNumber(), 0, 0, peek);

        if (m_Options.parseComplexMacros)
            m_Tokenizer.ReplaceFunctionLikeMacro(tk);
    }
}
But This is not true.
Code
bool Tokenizer::ReplaceFunctionLikeMacro(const Token* tk)
{
    wxString macroExpandedText;
    if ( GetMacroExpandedText(tk, macroExpandedText) )
        return ReplaceBufferText(macroExpandedText);
    return false;
}
Our recent changes in GetMacroExpandedText() function covers the handling of variable-like macro usage expansion.

So, the function name "ReplaceFunctionLikeMacro" cause confusing and need to be renamed.  ;D
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on October 02, 2014, 05:07:50 pm
...
On second look I noticed we can do a little optimization. This code:
Code
wxString arg = peek;
arg.Remove(0,1); // remove '('
if (arg.GetChar(0) == ParserConsts::ptr)
{
    arg.Remove(0,1).Trim(false); // remove '*'
    [...]
}
can be changed to:
Code
if (peek.GetChar(1) == ParserConsts::ptr)
{
    wxString arg = peek;
    arg.Remove(0,2).Trim(false); // remove "(*"
    [...]
}

So the entire code for this case:
Code
// see what is inside the (...)
// try to see whether the peek pattern is (* BBB)
if (peek.GetChar(1) == ParserConsts::ptr)
{
    wxString arg = peek;
    arg.RemoveLast(); // remove ")"
    arg.Remove(0,2).Trim(false); // remove "(* "
    m_Str << token;
    token = m_Tokenizer.GetToken(); //consume the peek
    // BBB is now the function ptr's name
    HandleFunction(/*function name*/ arg,
                   /*isOperator*/    false,
                   /*isPointer*/     true);
}
else if [...]
That way we don't have to create a temporary variable 'arg' every time when we see a function declaration-like pattern, but only when we know for sure it can be a function pointer.


Quote
I'm testing your patch now. It looks like macro expansion is enabled on every identifier like tokens.
Yes, when the (!switchHandled) case is reached in DoParse(), it now expands all macros, including variable-like ones.

Hi, Huki' patch is in trunk now, thanks for your contribution!
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on October 03, 2014, 03:36:18 pm
...
So, the function name "ReplaceFunctionLikeMacro" cause confusing and need to be renamed.  ;D
Done in the trunk r9934.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: WinterMute on December 04, 2014, 12:51:32 pm
Wow! Amazing job. Thank you so much for doing this - I really wasn't expecting it  :D
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on January 15, 2015, 09:44:42 pm
Hi,
I reviewed some remaining patches I have and noticed I have not yet submitted this one:

Oops yeah, I confirm, it works like in your screenshot, good job :) !

(Minor UI detail, it would be nice if the function calltip was showing the true name of the function and not PFNGLDRAWARRAYSINSTANCEDPROC)
Yes, I think it would be better to show the variable name rather than the typedef name. See here:
(http://i.imgur.com/UL1bCGl.png)

I'll submit the patch soon...

When we display the calltip for a typedef'd function pointer, we can show the function pointer's name rather than the typedef's name.
Here is the patch:

Code
From 65004d39e614519c66475a5295a6f6d44e4e3341 Mon Sep 17 00:00:00 2001
From: huki <gk7huki@gmail.com>
Date: Thu, 18 Sep 2014 18:25:20 +0530
Subject: CC: update typedef'd func pointer calltip

---
 src/plugins/codecompletion/nativeparser_base.cpp | 47 +++++++++++++-----------
 src/plugins/codecompletion/nativeparser_base.h   |  8 ++--
 2 files changed, 29 insertions(+), 26 deletions(-)

diff --git a/src/plugins/codecompletion/nativeparser_base.cpp b/src/plugins/codecompletion/nativeparser_base.cpp
index 778d55a..5abfadb 100644
--- a/src/plugins/codecompletion/nativeparser_base.cpp
+++ b/src/plugins/codecompletion/nativeparser_base.cpp
@@ -1685,7 +1685,7 @@ void NativeParserBase::ComputeCallTip(TokenTree*         tree,
 
             // either a function or a variable, but it is OK if a macro with not empty m_Args.
             if (tk && ((tk->m_TokenKind ^ tkMacroDef) || !tk->m_Args.empty()))
-                token = tk; // tkVariable could be a typedef'd function ptr (checked later down below)
+                token = tk; // tkVariable could be a typedef'd function ptr (checked in PrettyPrintToken())
             else
             {
                 // a variable like macro, this token don't have m_Args(call tip information), but
@@ -1711,33 +1711,36 @@ void NativeParserBase::ComputeCallTip(TokenTree*         tree,
             }
         }
 
-        // a variable basically don't have call tips, but if it's type is a typedef'd function
-        // pointer, we can still have call tips (which is the typedef function's arguments)
-        if (token->m_TokenKind == tkVariable)
-        {
-            const Token* tk = tree->at(tree->TokenExists(token->m_BaseType, token->m_ParentIndex, tkTypedef));
-            if (!tk && token->m_ParentIndex != -1)
-                tk = tree->at(tree->TokenExists(token->m_BaseType, -1, tkTypedef));
-            if (tk && !tk->m_Args.empty())
-                token = tk; // typedef'd function pointer
-        }
+        wxString tkTip;
+        if ( !PrettyPrintToken(tree, token, tkTip) )
+            tkTip = wxT("Error while pretty printing token!");
+        items.Add(tkTip);
 
-        {
-            wxString tkTip;
-            if ( !PrettyPrintToken(tree, token, tkTip) )
-                tkTip = wxT("Error while pretty printing token!");
-            items.Add(tkTip);
-        }
     }// for
 
     CC_LOCKER_TRACK_TT_MTX_UNLOCK(s_TokenTreeMutex)
 }
 
-bool NativeParserBase::PrettyPrintToken(const TokenTree*  tree,
-                                        const Token*      token,
-                                        wxString&         result,
-                                        bool              isRoot)
+bool NativeParserBase::PrettyPrintToken(TokenTree*      tree,
+                                        const Token*    token,
+                                        wxString&       result,
+                                        bool            isRoot)
 {
+    wxString name = token->m_Name;
+    // a variable basically don't have call tips, but if it's type is a typedef'd function
+    // pointer, we can still have call tips (which is the typedef function's arguments)
+    if (token->m_TokenKind == tkVariable)
+    {
+        const Token* tk = tree->at(tree->TokenExists(token->m_BaseType, token->m_ParentIndex, tkTypedef));
+        if (!tk && token->m_ParentIndex != -1)
+            tk = tree->at(tree->TokenExists(token->m_BaseType, -1, tkTypedef));
+        if (tk && !tk->m_Args.empty()) // typedef'd function pointer
+        {
+            name = token->m_Name;
+            token = tk;
+        }
+    }
+
     // if the token has parents and the token is a container or a function,
     // then pretty print the parent of the token->
     if (   (token->m_ParentIndex != -1)
@@ -1776,7 +1779,7 @@ bool NativeParserBase::PrettyPrintToken(const TokenTree*  tree,
             return true;
 
         case tkTypedef:
-            result = token->m_BaseType + wxT(" ") + result + token->m_Name + token->GetFormattedArgs();
+            result = token->m_BaseType + wxT(" ") + result + name + token->GetFormattedArgs();
             return true;
 
         case tkEnum:
diff --git a/src/plugins/codecompletion/nativeparser_base.h b/src/plugins/codecompletion/nativeparser_base.h
index 586c5ae..ab90894 100644
--- a/src/plugins/codecompletion/nativeparser_base.h
+++ b/src/plugins/codecompletion/nativeparser_base.h
@@ -321,10 +321,10 @@ protected:
     /** For ComputeCallTip()
      * No critical section needed in this recursive function!
      * All functions that call this recursive function, should already entered a critical section. */
-    bool PrettyPrintToken(const TokenTree*   tree,
-                          const Token*       token,
-                          wxString&          result,
-                          bool               isRoot = true);
+    bool PrettyPrintToken(TokenTree*    tree,
+                          const Token*  token,
+                          wxString&     result,
+                          bool          isRoot = true);
 
     // convenient static funcs for fast access and improved readability
 
--
1.9.4.msysgit.0



PS: I got your notification about checking macro usage for every token (http://forums.codeblocks.org/index.php/topic,19769.msg135047.html#msg135047), but I was too busy to check it earlier. Anyway, it's too complex so I think it's better to keep it aside for now...
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on February 23, 2015, 08:24:14 am
@Huki, sorry for the late reply, we are in spring festival now.  ;)
Happy Chinese new year to all guys!!!
You patch is in trunk now (rev 10124), thanks. I see that keeping the const qualifier in the function prototype needs a lot of work, so simply follow your way to remove the const.

About the macro replacement improvement, we have new serial patches, see: Macro expansion infinite loop. (http://forums.codeblocks.org/index.php/topic,19791.0.html). I tried to apply your patch on top of my macro replacement improvement serials, but the result is not expected as on trunk. Not sure why...

Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on April 06, 2015, 10:31:30 pm
Here is a patch attached, to support array of function pointer such as:
Code
int (*FuncArray[X][Y]) (...);

In the case of function pointers, we now call HandleFunction() with "name" argument in the form (*BBB) or (*BBB[X][Y]...). Then we strip the () * [] inside HandleFunction() to extract BBB.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on April 06, 2015, 11:57:43 pm
Some fixes in HandleTypedef() for typedef pointers:
When viewing tooltips for struct pointer typedefs (support added in r10155), we see a missing space and stray ")". From src/plugins/codecompletion/testing/cc_typedef_pointer.cpp:
Code
typedef struct foo1 * foo1Ptr;
hover on "foo1Ptr" and we see:
Code
typedef foo1foo1Ptr)

Fixed with this patch:
Code
@@ -3103,6 +3110,8 @@ bool ParserThread::ReadClsNames(wxString& ancestor)
             m_Str.clear();
             m_Str = ancestor;
 
+            m_PointerOrRef.Clear(); // FIX: for typedef pointers
+
             // Detects anonymous ancestor and gives him a name based on the first found alias.
             if (m_Str.StartsWith(g_UnnamedSymbol))
             {

The entire patch is attached below.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on April 08, 2015, 02:53:41 pm
About the macro replacement improvement, we have new serial patches, see: Macro expansion infinite loop. (http://forums.codeblocks.org/index.php/topic,19791.0.html). I tried to apply your patch on top of my macro replacement improvement serials, but the result is not expected as on trunk. Not sure why...
I'm currently testing the cc_macro serial patches and it's very stable so far. :) I'm fixing several bugs, the one mentioned in the above quote is fixed too.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on April 10, 2015, 03:26:27 am
About the macro replacement improvement, we have new serial patches, see: Macro expansion infinite loop. (http://forums.codeblocks.org/index.php/topic,19791.0.html). I tried to apply your patch on top of my macro replacement improvement serials, but the result is not expected as on trunk. Not sure why...
I'm currently testing the cc_macro serial patches and it's very stable so far. :) I'm fixing several bugs, the one mentioned in the above quote is fixed too.
Hi, thanks for the testing!
I just tested your two patches.
1, patch handling array of function pointer, this works fine! I also add the test case to our cctest.
2, patch handling typedef pointers. I have some questions:

The token is like below before and after your patch:
Code
before

$3 = {m_FullType = L"foo1*", m_BaseType = L"foo1", m_Name = L"foo1Ptr", m_Args = <g_strEmpty+12> L"", m_BaseArgs = <g_strEmpty+12> L"", m_AncestorsString = <g_strEmpty+12> L"", m_FileIdx = 1, m_Line = 9, m_ImplFileIdx = 0, m_ImplLine = 0, m_ImplLineStart = 0, m_ImplLineEnd = 0, m_Scope = tsUndefined, m_TokenKind = tkTypedef, m_IsOperator = false, m_IsLocal = true, m_IsTemp = false, m_IsConst = false, m_IsNoExcept = false, m_IsAnonymous = false, m_Index = 248, m_ParentIndex = -1
, m_Children = std::set with 0 elements, m_Ancestors = std::set with 2 elements = {[0] = 245, [1] = 247}, m_DirectAncestors = std::set with 2 elements = {[0] = 245, [1] = 247}, m_Descendants = std::set with 0 elements, m_Aliases = wxArray<T>, m_TemplateArgument = <g_strEmpty+12> L"", m_TemplateType = wxArray<T>, m_TemplateMap = std::map with 0 elements, m_TemplateAlias = <g_strEmpty+12> L"", m_UserData = 0x0, m_TokenTree = 0x9a7d020, m_Ticket = 504}


After

$1 = {m_FullType = L"foo1"
, m_BaseType = L"foo1", m_Name = L"foo1Ptr", m_Args = <g_strEmpty+12> L"", m_BaseArgs = <g_strEmpty+12> L"", m_AncestorsString = <g_strEmpty+12> L"", m_FileIdx = 1, m_Line = 9, m_ImplFileIdx = 0, m_ImplLine = 0, m_ImplLineStart = 0, m_ImplLineEnd = 0, m_Scope = tsUndefined, m_TokenKind = tkTypedef, m_IsOperator = false, m_IsLocal = true, m_IsTemp = false, m_IsConst = false, m_IsNoExcept = false, m_IsAnonymous = false, m_Index = 248, m_ParentIndex = -1, m_Children = std::set with 0 elements, m_Ancestors = std::set with 2 elements = {[0] = 245, [1] = 247}, m_DirectAncestors = std::set with 2 elements = {[0] = 245, [1] = 247}, m_Descendants = std::set with 0 elements, m_Aliases = wxArray<T>, m_TemplateArgument = <g_strEmpty+12> L""
, m_TemplateType = wxArray<T>, m_TemplateMap = std::map with 0 elements, m_TemplateAlias = <g_strEmpty+12> L""
, m_UserData = 0x0, m_TokenTree = 0x9a6aa20, m_Ticket = 504}


Do you think we need to remove the "*" from the "m_FullType"?

I just debugged a little, and I found this code snippet:

Code
wxString Token::DisplayName() const
{
    wxString result;
    if      (m_TokenKind == tkClass)
        return result << _T("class ")     << m_Name << m_BaseArgs << _T(" {...}");
    else if (m_TokenKind == tkNamespace)
        return result << _T("namespace ") << m_Name << _T(" {...}");
    else if (m_TokenKind == tkEnum)
        return result << _T("enum ")      << m_Name << _T(" {...}");
    else if (m_TokenKind == tkTypedef)
    {
        result << _T("typedef");

        if (!m_FullType.IsEmpty())
            result << _T(" ") << m_FullType;

        if (result.Find('*', true) != wxNOT_FOUND)
        {
            result.RemoveLast();
            return result << m_Name << _T(")") <<  GetFormattedArgs();
        }

        if (!m_TemplateArgument.IsEmpty())
            result << m_TemplateArgument;

        return result << _T(" ") << m_Name;
    }
Question is:
What does the below code's purpose?
Code
        if (result.Find('*', true) != wxNOT_FOUND)
        {
            result.RemoveLast();
            return result << m_Name << _T(")") <<  GetFormattedArgs();
        }

Thanks.

Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: Huki on April 10, 2015, 08:34:47 pm
Do you think we need to remove the "*" from the "m_FullType"?
It was the only way I could fix the "typedef foo1foo1Ptr)" tooltip bug, but now you have found the real source of the bug in Token::DisplayName(), so we don't have to  remove the "*" anymore (see below).

Question is:
What does the below code's purpose?
Code
        if (result.Find('*', true) != wxNOT_FOUND)
        {
            result.RemoveLast();
            return result << m_Name << _T(")") <<  GetFormattedArgs();
        }

In HandleTypedef, we support 2 cases of typedef'd function pointers, and in each case the type is stored as below:
Code
typedef void (*dMessageFunction)(int errnum, const char *msg, va_list ap);
// --> type is stored as: (*)

typedef void (MyClass::*Function)(int);
// --> type is stored as: (MyClass::*)

Then in Token::DisplayName() this type is taken as "result". So the code below removes the last ')' and adds the m_Name:
Code
        // result is "typedef void (*)", m_Name is "Function"
        if (result.Find('*', true) != wxNOT_FOUND)
        {
            result.RemoveLast(); // result is "typedef void (*"
            return result << m_Name << _T(")") <<  GetFormattedArgs(); // we return "typedef void (*Function) (args)"
        }

So the bug occurs when we test for "if (result.Find('*', true) != wxNOT_FOUND)", but the typedef is not a function pointer (it could be a struct pointer).
A way to fix it is to ensure we really have ')' as the last char. I suggest changing the test like this:
Code
if (result.Find('*', true) != wxNOT_FOUND && result.Last() == ')')

Then my previous hack to clear m_PointerOrRef is not required.. :) I quickly tested it and it's working: I get "typedef foo1* foo1Ptr" as expected.
Title: Re: Is it possible for the parser to support newlib prototypes?
Post by: ollydbg on April 11, 2015, 02:42:30 pm
Hi, Huki, thanks for the detailed explanation, the modified patches is in trunk now. (I also added two test cases for our CCTest project)