Author Topic: Macro expansion infinite loop.  (Read 28037 times)

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13438
    • Travis build status
Macro expansion infinite loop.
« on: November 30, 2014, 05:06:36 pm »
I've reported the problem here:
http://forums.codeblocks.org/index.php/topic,19753.msg134925.html#msg134925
http://forums.codeblocks.org/index.php/topic,19753.msg134942.html#msg134942

And I've debugged it a bit more and I know what code pattern breaks it or at least this is what cause it to loop.
You can see it in the following workspace http://cmpt.benbmp.org/codeblocks/nightlies/cc_marco_infinite.zip
Unfortunately when this is extracted from the original code it doesn't have the same effect.
So this project cannot be used to reproduce the problem. :(

ollydbg: Can you look at the project and see if it can give you some hints what is going wrong with this code?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5380
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Macro expansion infinite loop.
« Reply #1 on: December 01, 2014, 08:54:36 am »
ollydbg: Can you look at the project and see if it can give you some hints what is going wrong with this code?
I have tried your test workspace, but I don't get much hints. :(
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13438
    • Travis build status
Re: Macro expansion infinite loop.
« Reply #2 on: December 01, 2014, 03:06:49 pm »
Any hints how to debug it further or how to find what is the thing that causes the problem?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5380
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Macro expansion infinite loop.
« Reply #3 on: December 03, 2014, 03:31:02 am »
Any hints how to debug it further or how to find what is the thing that causes the problem?
I have an idea. You can check how many times the function DoAddToken() was called. So, just set a breakpoint in the function, and count the times, if it get a very large number, there indicates a bad situation.  E.g. If you workspace contains about 10000 tokens(this can be seen from an old version of C::B which don't cause the infinite loop), and you see DoAddToken() has been called 20000 times in the newer C::B, then it may be in an infinite loop now, and you can step in and see where the infinite loop are.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13438
    • Travis build status
Re: Macro expansion infinite loop.
« Reply #4 on: January 23, 2015, 12:39:17 pm »
OK, I've spend some more time and now I was able to make a project that reproduces the problem.

Here it is http://cmpt.benbmp.org/codeblocks/nightlies/cc_marco_infinite2.tar.gz

ollydbg: Can you try to see if you can reproduce and fix the problem?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5380
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Macro expansion infinite loop.
« Reply #5 on: January 23, 2015, 01:02:03 pm »
OK. I will test it.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5380
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Macro expansion infinite loop.
« Reply #6 on: January 23, 2015, 03:16:19 pm »
I see infinite loop too. :(

Bug located. Patch here:
Code
 src/plugins/codecompletion/parser/tokenizer.cpp | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/plugins/codecompletion/parser/tokenizer.cpp b/src/plugins/codecompletion/parser/tokenizer.cpp
index 764bcbe..6202eda 100644
--- a/src/plugins/codecompletion/parser/tokenizer.cpp
+++ b/src/plugins/codecompletion/parser/tokenizer.cpp
@@ -1794,6 +1794,10 @@ bool Tokenizer::ReplaceBufferText(const wxString& target)
             m_TokenIndex = m_BufferLen - m_FirstRemainingLength;
             m_PeekAvailable = false;
             SkipToEOL(false);
+            SkipToOneOfChars(_T(",;}"), true, false, false);
+            m_SavedTokenIndex   = m_UndoTokenIndex = m_TokenIndex;
+            m_SavedLineNumber   = m_UndoLineNumber = m_LineNumber;
+            m_SavedNestingLevel = m_UndoNestLevel  = m_NestLevel;
             return false;
         }
         else
It is a workaround patch and will cause parsing the Test class body wrongly, you will notice one extra member in the symbol tree :). But it works around a bug that macro replacement cause infinite loop when parsing. The fundamental solution is to ONLY do macro replacement in the Tokenizer level. (We currently do in both ParserThread and Tokenizer).

Can you test it?

I noticed another issue is that token numbers are becomes large because we save all the "macro usage tokens". I think they should not be saved in the tokentree, they are just like function call. Since we only restore the function declaration/definition, we don't save the function call, I think we also don't need to store the macro usage.

About macro usage check in the Tokenizer level, I have locally implement this feature, and it solve the bug Macro replacement in CC should remember the used macro definition(this is the similar issue as OBF's test case), but my local implementation contains many commits and massive code change in Tokenizer and Parserthread class. :(


« Last Edit: January 23, 2015, 03:18:50 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13438
    • Travis build status
Re: Macro expansion infinite loop.
« Reply #7 on: January 23, 2015, 07:35:18 pm »
Yes, it does prevent the infinite loop and yes it screws the members of the struct that are declared after the broken member.

If you have some code that needs testing post a branch on github or even a patch and I'll give it a try.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5380
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Macro expansion infinite loop.
« Reply #8 on: January 24, 2015, 02:45:42 pm »
Yes, it does prevent the infinite loop and yes it screws the members of the struct that are declared after the broken member.

If you have some code that needs testing post a branch on github or even a patch and I'll give it a try.
OK, after taking several hours to clean up my local git(merge some patches, split some patches, move some patches up and down) , I create a patches serials which contains 31 patches which are against SVN rev 10081.
Any comments are welcome.  :)


If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9615
Re: Macro expansion infinite loop.
« Reply #9 on: January 25, 2015, 12:40:07 pm »
Any comments are welcome.  :)
No patches against SVN, and/or does not apply against SVN, so I am out (as usual)...
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13438
    • Travis build status
Re: Macro expansion infinite loop.
« Reply #10 on: January 25, 2015, 07:44:22 pm »
Morten: Check this branch out https://github.com/obfuscated/codeblocks_sf/tree/ollydbg/cc_macro it has all  the commits applied.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13438
    • Travis build status
Re: Macro expansion infinite loop.
« Reply #11 on: January 25, 2015, 09:52:59 pm »
@ollydbg:
I don't see any patches that add tests.
Am I blind or this is the case (and why)?

@Morten: There is a link for svn friendly (hope so) patch http://cmpt.benbmp.org/codeblocks/patches/cc.marco.patch
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5380
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Macro expansion infinite loop.
« Reply #12 on: January 26, 2015, 03:15:24 am »
@ollydbg:
I don't see any patches that add tests.
Am I blind or this is the case (and why)?
I will add one(new cpp file) to the src\plugins\codecompletion\testing folder to test the Macro expansion infinite loop issue. You will need to build the CCTest project to run all the tests.

Quote
@Morten: There is a link for svn friendly (hope so) patch http://cmpt.benbmp.org/codeblocks/patches/cc.marco.patch
Thanks, OBF.
@Morten, this is a merged SVN style patch containing all my 31 Git style patches. I think it is too hard for reviewing such a single patch.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9615
Re: Macro expansion infinite loop.
« Reply #13 on: January 27, 2015, 08:31:18 pm »
Quote
@Morten: There is a link for svn friendly (hope so) patch http://cmpt.benbmp.org/codeblocks/patches/cc.marco.patch
Thanks, OBF.
@Morten, this is a merged SVN style patch containing all my 31 Git style patches. I think it is too hard for reviewing such a single patch.
Oh yes, this one works. :-) I can start testing...
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13438
    • Travis build status
Re: Macro expansion infinite loop.
« Reply #14 on: January 27, 2015, 09:08:51 pm »
I'm testing it for a second day and I don't see major problems (crashes, infinite loops).
I can't comment much about the quality of the parser.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]