Author Topic: A bug in handling #define  (Read 13885 times)

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5721
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
A bug in handling #define
« on: February 06, 2010, 06:45:14 am »
For example, when parsing this statement:
Code
#define _NO_DEBUG_PLACEHOLDER _No_debug_placeholder = _No_debug_placeholder()

See the source code in parserthread.cpp
Code
void ParserThread::HandleDefines()
{
    wxString filename;
    int lineNr = m_Tokenizer.GetLineNumber();
    wxString token = m_Tokenizer.GetToken(); // read the token after #define
    m_Str.Clear();
    // now token holds something like:
    // BLAH_BLAH
    if (!token.IsEmpty())
    {
        // skip the rest of the #define
        wxString defVal = token + m_Tokenizer.ReadToEOL();
        wxString para(_T(""));
        int start = defVal.Find('(');
        int end   = defVal.Find(')');

        TRACE(_T("HandleDefines() : Saving nesting level: %d,%d"), start, end);

        // make sure preprocessor definitions are not going under namespaces or classes!
        if (start != wxNOT_FOUND && end != wxNOT_FOUND)
        {
            para = defVal.Mid(start, end-start+1);
            m_Str = defVal.Mid(end + 1);
            m_Str.Trim(false);
        }
        else
        {
            m_Str = defVal.substr(token.length());
            m_Str.Trim(false);
            //defVal = _T("");
        }
        Token* oldParent = m_pLastParent;
        m_pLastParent = 0L;
        DoAddToken(tkPreprocessor, token, lineNr, lineNr, m_Tokenizer.GetLineNumber(), para, false, true);
        m_pLastParent = oldParent;
        m_Str.Clear();
    }
}

The "()" at the end of the line was mistakenly regarded as the "parameter" of the define, like below:
Code
#define _NO_DEBUG_PLACEHOLDER() XXXXXX
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: A bug in handling #define
« Reply #1 on: February 17, 2010, 01:35:49 pm »
Patch for it:
« Last Edit: February 19, 2010, 06:43:24 am by blueshake »
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5721
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: A bug in handling #define
« Reply #2 on: February 17, 2010, 01:55:41 pm »
thank you.now, i'm at home and view this forum in my gphone. i will test it when i am at work.
Happy Chinese New Year to every fourmers!
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: A bug in handling #define
« Reply #3 on: February 18, 2010, 07:59:30 am »
it is better to replace
Code
m_Str << m_Tokenizer.ReadToEOL();

with
Code
if (!m_Str.IsEmpty())
    m_Str << _T(" ") <<  m_Tokenizer.ReadToEOL();

we need a space between two word here.

Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9660
Re: A bug in handling #define
« Reply #4 on: February 18, 2010, 09:14:43 am »
Code
if (!m_Str.IsEmpty())
    m_Str << _T(" ") <<  m_Tokenizer.ReadToEOL();
we need a space between two word here.
Are you sure? Because in the case m_Str.IsEmpty(), m_Tokenizer.ReadToEOL() is not being called. Didn't you mean something like:
Code
if (!m_Str.IsEmpty())
    m_Str << _T(" "):
m_Str << m_Tokenizer.ReadToEOL();
???
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: A bug in handling #define
« Reply #5 on: February 18, 2010, 09:29:12 am »
haha,you are right. :D

Code
if (!m_Str.IsEmpty())
    m_Str << _T(" ");
m_Str << m_Tokenizer.ReadToEOL();
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9660
Re: A bug in handling #define
« Reply #6 on: February 18, 2010, 03:32:46 pm »
haha,you are right. :D
Probably this or the other patch of yours makes the total number of tokens to decrease from 138000 to 119000 in my C::B project. This doesn't look good to me.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9660
Re: A bug in handling #define
« Reply #7 on: February 18, 2010, 03:35:39 pm »
This doesn't look good to me.
It seems that all "simple" defines like:
#define BOOST_ALIGNED_STORAGE_HPP
are lost now.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: A bug in handling #define
« Reply #8 on: February 19, 2010, 03:09:42 am »
this one cause the issue.I will check into it.
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: A bug in handling #define
« Reply #9 on: February 19, 2010, 03:22:21 am »
Ok,it seems I have solve this.I will update it when I get my machine.
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: A bug in handling #define
« Reply #10 on: February 19, 2010, 04:16:12 am »
new patch:


I just add a line compare here.because for codes:

Code
#define BOOST_ALIGNED_STORAGE_HPP
int main() {}

when we do this,
Code
wxString para = m_Tokenizer.GetToken();

at this time ,m_Tokenizer will return int ,then we do this
Code
m_Str << m_Tokenizer.ReadToEOL();

it will eat the whole line codes
Code
int main()

that is why the tokens number are lost.



BTW. @morten

can we print the whole project's tokens name to the file??so when we make patch,when the tokens numbers are wrong.we can use diff tool to check whick tokens are lost.




« Last Edit: February 19, 2010, 06:43:04 am by blueshake »
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: A bug in handling #define
« Reply #11 on: February 19, 2010, 06:42:41 am »
Another update.

remove the compile warning.
forget to do this in last patch.
Quote
if (!m_Str.IsEmpty())
    m_Str << _T(" ");
m_Str << m_Tokenizer.ReadToEOL();


Code
Index: src/plugins/codecompletion/parser/parserthread.cpp
===================================================================
--- src/plugins/codecompletion/parser/parserthread.cpp (revision 6169)
+++ src/plugins/codecompletion/parser/parserthread.cpp (working copy)
@@ -1123,8 +1123,8 @@
 
 void ParserThread::HandleDefines()
 {
-    wxString filename;
-    int lineNr = m_Tokenizer.GetLineNumber();
+    //wxString filename;
+    size_t lineNr = m_Tokenizer.GetLineNumber();
     wxString token = m_Tokenizer.GetToken(); // read the token after #define
     m_Str.Clear();
     // now token holds something like:
@@ -1132,31 +1132,30 @@
     if (!token.IsEmpty())
     {
         // skip the rest of the #define
-        wxString defVal = token + m_Tokenizer.ReadToEOL();
-        wxString para(_T(""));
-        int start = defVal.Find('(');
-        int end   = defVal.Find(')');
-
+        wxString para = m_Tokenizer.GetToken();
         TRACE(_T("HandleDefines() : Saving nesting level: %d,%d"), start, end);
-
-        // make sure preprocessor definitions are not going under namespaces or classes!
-        if (start != wxNOT_FOUND && end != wxNOT_FOUND)
+        if (lineNr == m_Tokenizer.GetLineNumber())
         {
-            para = defVal.Mid(start, end-start+1);
-            m_Str = defVal.Mid(end + 1);
-            m_Str.Trim(false);
+            if (para.IsEmpty() || para.GetChar(0) != '(') //check if contain the open brace
+            {
+                m_Str = para;
+                para = wxEmptyString;
+            }
+            if (!m_Str.IsEmpty())
+                m_Str << _T(" ");
+            m_Str << m_Tokenizer.ReadToEOL();
         }
         else
         {
-            m_Str = defVal.substr(token.length());
-            m_Str.Trim(false);
-            //defVal = _T("");
+            m_Tokenizer.UngetToken();
+            para = wxEmptyString;
+            m_Str << m_Tokenizer.ReadToEOL();
         }
         Token* oldParent = m_pLastParent;
         m_pLastParent = 0L;
         DoAddToken(tkPreprocessor, token, lineNr, lineNr, m_Tokenizer.GetLineNumber(), para, false, true);
         m_pLastParent = oldParent;
-        m_Str.Clear();
+        //m_Str.Clear();
     }
 }
 
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9660
Re: A bug in handling #define
« Reply #12 on: February 19, 2010, 06:45:48 am »
can we print the whole project's tokens name to the file??so when we make patch,when the tokens numbers are wrong.we can use diff tool to check whick tokens are lost. [/b]
Sure. Using the token debug window (CTRL + double-click on a token). There you'll find a save  button that allows to save the file list as well as the whole token tree to a file. That's why I could tell you what's missing btw... ;-)

But be careful: Creating the token tree can take a long time! C::B seems frozen then but it's not. Just be patient.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9660
Re: A bug in handling #define
« Reply #13 on: February 19, 2010, 06:47:30 am »
at this time ,m_Tokenizer will return int ,then we do this
Code
m_Str << m_Tokenizer.ReadToEOL();
It depends on the settings of the Tokenizer (what to skip) IMHO.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ