Author Topic: The 08 February 2014 build (9639) is out.  (Read 38717 times)

Offline kingfox

  • Multiple posting newcomer
  • *
  • Posts: 41
Re: The 08 February 2014 build (9639) is out.
« Reply #15 on: February 11, 2014, 01:19:07 pm »
"new command line option --user-data-dir=<path>" is very good !  :D

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5247
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: The 08 February 2014 build (9639) is out.
« Reply #16 on: February 11, 2014, 01:33:36 pm »
What I tried is to create a new wxWidgets project, without PCH, a debug configuration (which uses the wxWidgets 3.0.0 release libs) and a release configuration. The project contains no code changes, only the generated files. After opening the project, the CPU load increases up to 100% and CB can only be killed via Task Manager.

Here is the content of the project file (I don't want to provide any zip archives):

Code: [Select]
?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_project_file>
<FileVersion major="1" minor="6" />
<Project>
<Option title="minimal_test" />
<Option pch_mode="2" />
<Option compiler="gcc" />
<Build>
<Target title="Debug">
<Option output="bin/minimal_test" prefix_auto="1" extension_auto="1" />
<Option object_output="bin/" />
<Option type="0" />
<Option compiler="gcc" />
<Option projectLinkerOptionsRelation="2" />
<Compiler>
<Add option="-g" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</Compiler>
<ResourceCompiler>
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</ResourceCompiler>
<Linker>
<Add library="libwxmsw30u_core.a" />
<Add library="libwxbase30u.a" />
<Add library="libwxpng.a" />
<Add library="libwxzlib.a" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib" />
</Linker>
</Target>
<Target title="Release">
<Option output="bin/minimal_test" prefix_auto="1" extension_auto="1" />
<Option object_output="bin/" />
<Option type="0" />
<Option compiler="gcc" />
<Option projectLinkerOptionsRelation="2" />
<Compiler>
<Add option="-O2" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</Compiler>
<ResourceCompiler>
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</ResourceCompiler>
<Linker>
<Add option="-s" />
<Add library="libwxmsw30u_core.a" />
<Add library="libwxbase30u.a" />
<Add library="libwxpng.a" />
<Add library="libwxzlib.a" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib" />
</Linker>
</Target>
</Build>
<Compiler>
<Add option="-pipe" />
<Add option="-mthreads" />
<Add option="-D__GNUWIN32__" />
<Add option="-D__WXMSW__" />
<Add option="-DwxUSE_UNICODE" />
<Add option="-Wall" />
<Add directory="C:/Libs/wxWidgets_gcc/include" />
</Compiler>
<ResourceCompiler>
<Add directory="C:/Libs/wxWidgets_gcc/include" />
</ResourceCompiler>
<Linker>
<Add option="-mthreads" />
<Add library="libkernel32.a" />
<Add library="libuser32.a" />
<Add library="libgdi32.a" />
<Add library="libwinspool.a" />
<Add library="libcomdlg32.a" />
<Add library="libadvapi32.a" />
<Add library="libshell32.a" />
<Add library="libole32.a" />
<Add library="liboleaut32.a" />
<Add library="libuuid.a" />
<Add library="libcomctl32.a" />
<Add library="libwsock32.a" />
<Add library="libodbc32.a" />
</Linker>
<Unit filename="minimal_testApp.cpp" />
<Unit filename="minimal_testApp.h" />
<Unit filename="minimal_testMain.cpp" />
<Unit filename="minimal_testMain.h" />
<Unit filename="resource.rc">
<Option compilerVar="WINDRES" />
</Unit>
<Extensions>
<code_completion />
<envvars />
<debugger />
<lib_finder disable_auto="1" />
</Extensions>
</Project>
</CodeBlocks_project_file>


OK, many thanks. I can reproduce the hang issue with the latest C::B SVN head, so I will try to see what cause this hang issue.

Here is the BT that I see m_TokenIndex always stay in the same value (infinite loop)
Code: [Select]
[debug]#0  Tokenizer::ReplaceMacro (this=0x940fc98, str="wxDEFINE_UNICHARREF_OPERATOR") at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\tokenizer.cpp:1259
[debug]#1  0x65ef64b1 in Tokenizer::DoGetToken (this=0x940fc98) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\tokenizer.cpp:1230
[debug]#2  0x65ef5e19 in Tokenizer::PeekToken (this=0x940fc98) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\tokenizer.cpp:1064
[debug]#3  0x65ee3ba9 in ParserThread::DoParse (this=0x940fc90) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.cpp:1002
[debug]#4  0x65ee7107 in ParserThread::HandleClass (this=0x940fc90, ct=ParserThread::ctClass) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.cpp:1971
[debug]#5  0x65ee3262 in ParserThread::DoParse (this=0x940fc90) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.cpp:811
[debug]#6  0x65ee23df in ParserThread::Parse (this=0x940fc90) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.cpp:513
[debug]#7  0x65f0607e in ParserThread::Execute (this=0x940fc90) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.h:155
[debug]#8  0x010ab44b in cbThreadPool::cbWorkerThread::Entry (this=0x6bab438) at F:\cb_sf_git\trunk\src\sdk\cbthreadpool.cpp:216
[debug]#9  0x62764ba9 in wxThreadInternal::DoThreadStart(wxThread*) () from E:\code\wx-mingw-build-481-dw2\wxWidgets-2.8.12\lib\gcc_dll\wxmsw28u_gcc_custom.dll
[debug]#10 0x62764c85 in wxThreadInternal::WinThreadStart(void*)@4 () from E:\code\wx-mingw-build-481-dw2\wxWidgets-2.8.12\lib\gcc_dll\wxmsw28u_gcc_custom.dll
[debug]#11 0x77c3a3b0 in msvcrt!_endthreadex () from C:\WINDOWS\system32\msvcrt.dll
[debug]#12 0x7c80b729 in KERNEL32!GetModuleFileNameA () from C:\WINDOWS\system32\kernel32.dll
[debug]#13 0x00000000 in ?? ()
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9508
Re: The 08 February 2014 build (9639) is out.
« Reply #17 on: February 11, 2014, 04:24:40 pm »
OK, many thanks. I can reproduce the hang issue with the latest C::B SVN head
I can do as well, meanwhile with all my wx30 based projects... dammed.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9508
Re: The 08 February 2014 build (9639) is out.
« Reply #18 on: February 11, 2014, 09:14:03 pm »
Here is the BT that I see m_TokenIndex always stay in the same value (infinite loop)
For me its different: It looks like in bool ParserThread::Parse() in the else if (!switchHandled) clause it always end in the m_Str << token << ParserConsts::space_chr; line which will crash sooner or later if the wxString cannot append more characters. at that time, token is something like (wxMACRO(...)) which cannot be resolved / handled in this loop.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline blauzahn

  • Multiple posting newcomer
  • *
  • Posts: 101
Re: The 08 February 2014 build (9639) is out.
« Reply #19 on: February 12, 2014, 12:15:33 am »
a superficial glance for typical suspects shows:

class Token's ctor should initialize its members to avoid UB:

m_IsConst(false),
m_IsNoExcept(false),

I see no reason why its data members m_TokenTree and m_ticket have to be protected instead of private.
Is this class still in use?

class Tokenizer's ctor looks like it better should be explicit. Its 2nd argument has a default value and an
accidential implicit construction out of a TokenTree* feels quite unsound to me.

Offline blauzahn

  • Multiple posting newcomer
  • *
  • Posts: 101
Re: The 08 February 2014 build (9639) is out.
« Reply #20 on: February 12, 2014, 12:27:29 am »
For me its different: It looks like in bool ParserThread::Parse() in the else if (!switchHandled) clause it always end in the m_Str << token << ParserConsts::space_chr; line which will crash sooner or later if the wxString cannot append more characters. at that time, token is something like (wxMACRO(...)) which cannot be resolved / handled in this loop.
You mean void ParserThread::DoParse(), I assume.

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: The 08 February 2014 build (9639) is out.
« Reply #21 on: February 12, 2014, 02:16:36 am »
"new command line option --user-data-dir=<path>" is very good !  :D

Good! Let us know if there are issues.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5247
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: The 08 February 2014 build (9639) is out.
« Reply #22 on: February 12, 2014, 03:51:52 am »
Here is the BT that I see m_TokenIndex always stay in the same value (infinite loop)
For me its different: It looks like in bool ParserThread::Parse() in the else if (!switchHandled) clause it always end in the m_Str << token << ParserConsts::space_chr; line which will crash sooner or later if the wxString cannot append more characters. at that time, token is something like (wxMACRO(...)) which cannot be resolved / handled in this loop.

I debugged a little, I think it was a regression after a patch by Huki, that is:

Code: [Select]
Revision: f7cb07c8dcf613f6786eaf0ee34e908ec218d2c8
Author: ollydbg <[email protected]>
Date: 2013-9-29 16:25:10
Message:
* CC: reliable working of UngetToken() when macro expansion is involved, it set the undo index value correctly(thanks Huki)

* CC: avoid take backward step (UngetToken) twice in the Tokenizer class
- CC: comments added for Tokenizer class

git-svn-id: https://svn.code.sf.net/p/codeblocks/code/[email protected] 2a5c6006-c6dd-42ca-98ab-0921f2732cef
----
Modified: src/plugins/codecompletion/parser/tokenizer.cpp
Modified: src/plugins/codecompletion/parser/tokenizer.h


which have such diff:
Code: [Select]
@@ -1148,6 +1148,8 @@ wxString Tokenizer::PeekToken()
         unsigned int savedLineNumber = m_LineNumber;
         unsigned int savedNestLevel  = m_NestLevel;
 
+        int savedReplaceCount = m_IsReplaceParsing ? m_RepeatReplaceCount : -1;
+
         if (SkipUnwanted())
             m_PeekToken = DoGetToken();
         else
@@ -1156,17 +1158,33 @@ wxString Tokenizer::PeekToken()
         m_PeekTokenIndex             = m_TokenIndex;
         m_PeekLineNumber             = m_LineNumber;
         m_PeekNestLevel              = m_NestLevel;
-
-        m_TokenIndex                 = savedTokenIndex;
-        m_LineNumber                 = savedLineNumber;
-        m_NestLevel                  = savedNestLevel;
+        // Check whether a ReplaceBufferForReparse() was done in DoGetToken().
+        // We assume m_Undo... have already been reset in ReplaceBufferForReparse().
+        if (m_IsReplaceParsing && savedReplaceCount != (int)m_RepeatReplaceCount)
+        {
+            m_TokenIndex             = m_UndoTokenIndex;
+            m_LineNumber             = m_UndoLineNumber;
+            m_NestLevel              = m_UndoNestLevel;
+        }
+        else
+        {
+            m_TokenIndex             = savedTokenIndex;
+            m_LineNumber             = savedLineNumber;
+            m_NestLevel              = savedNestLevel;
+        }
     }
 
     return m_PeekToken;

So, bug is here:
Code: [Select]
wxString Tokenizer::PeekToken()
{
    if (!m_PeekAvailable)
    {
        m_PeekAvailable = true;

        unsigned int savedTokenIndex = m_TokenIndex;
        unsigned int savedLineNumber = m_LineNumber;
        unsigned int savedNestLevel  = m_NestLevel;

        int savedReplaceCount = m_IsReplaceParsing ? m_RepeatReplaceCount : -1;

        if (SkipUnwanted())
            m_PeekToken = DoGetToken();
        else
            m_PeekToken.Clear();

        m_PeekTokenIndex             = m_TokenIndex;
        m_PeekLineNumber             = m_LineNumber;
        m_PeekNestLevel              = m_NestLevel;
        // Check whether a ReplaceBufferText() was done in DoGetToken().
        // We assume m_Undo... have already been reset in ReplaceBufferText().
        if (m_IsReplaceParsing && savedReplaceCount != (int)m_RepeatReplaceCount)
        {
            m_TokenIndex             = m_UndoTokenIndex;                       //*************bug here******************
            m_LineNumber             = m_UndoLineNumber;
            m_NestLevel              = m_UndoNestLevel;
        }
        else
        {
            m_TokenIndex             = savedTokenIndex;
            m_LineNumber             = savedLineNumber;
            m_NestLevel              = savedNestLevel;
        }
    }

    return m_PeekToken;
}

m_TokenIndex is always remain/reset to the same value, so Tokenizer won't go forward any more.


If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline blauzahn

  • Multiple posting newcomer
  • *
  • Posts: 101
Re: The 08 February 2014 build (9639) is out.
« Reply #23 on: February 12, 2014, 07:00:42 am »
Has it been considered to group

TokenIndex
LineNumber
NestLevel

into a tiny class/struct?

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5247
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: The 08 February 2014 build (9639) is out.
« Reply #24 on: February 12, 2014, 07:14:20 am »
Has it been considered to group

TokenIndex
LineNumber
NestLevel

into a tiny class/struct?
Hi, blauzahn, thanks. Sure this will make the code more clean and readable(I even thought the lexeme string should be bundled with those information to a single struct), but the change is not simple, because two many places use those variables. :)
I'm currently don't have the time to do that, so this can be a TO-DO.
Let's firstly fix the hang issue. ;)
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9508
Re: The 08 February 2014 build (9639) is out.
« Reply #25 on: February 12, 2014, 08:16:13 am »
m_TokenIndex is always remain/reset to the same value, so Tokenizer won't go forward any more.
That's too bad. Can you fix it? Otherwise I would strongly recommend to revert this change completely - it makes C::B unusable with wx30.

Edit: What SVN revision this belongs to, btw?! ???
« Last Edit: February 12, 2014, 08:20:30 am by MortenMacFly »
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5247
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: The 08 February 2014 build (9639) is out.
« Reply #26 on: February 12, 2014, 08:27:38 am »
m_TokenIndex is always remain/reset to the same value, so Tokenizer won't go forward any more.
That's too bad. Can you fix it? Otherwise I would strongly recommend to revert this change completely - it makes C::B unusable with wx30.

Edit: What SVN revision this belongs to, btw?! ???

Follow this post:
Re: Several improvements to Code Completion plugin
and
Re: Several improvements to Code Completion plugin

The change is rev 9369, you can see it from the git log message in my previous posts.
Quote
Revision: f7cb07c8dcf613f6786eaf0ee34e908ec218d2c8
Author: ollydbg <[email protected]>
Date: 2013-9-29 16:25:10
Message:
* CC: reliable working of UngetToken() when macro expansion is involved, it set the undo index value correctly(thanks Huki)

* CC: avoid take backward step (UngetToken) twice in the Tokenizer class
- CC: comments added for Tokenizer class

git-svn-id: https://svn.code.sf.net/p/codeblocks/code/[email protected]9369 2a5c6006-c6dd-42ca-98ab-0921f2732cef

Currently I can't fix it, but I'm debugging now for hours......
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5247
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: The 08 February 2014 build (9639) is out.
« Reply #27 on: February 12, 2014, 09:27:41 am »
The code do the macro replacement is a mass, the big error looks like it is not the way a C Processor do the replacement.

For example, if we have such code

Code: [Select]
/****************************************/

#define A(x,y)  (x+y)
#define B 1
#define C 2


#if A(B,C)

void f1234();

#endif

void f2345();


// f123      //f1234
// f234      //f2345
To expand the A(B,C), we should first apply the macro definition 1, so it becomes (B+C), then we should "rescan" the result token list, and finally get the (2+1), so the final result is 3.

But from my point of view, our Tokenizer don't have a "rescan" stage, the trick thing is that it did some text substitute on formal parameter to actual parameter translation, some kind of recursive call.

Note, with the latest cc-test frame, I get such result
Code: [Select]
-PASS:  f234  f2345
*FAIL:  f123  f1234
--------------------------------------------------------
Total 2 tests, 1 PASS, 1 FAIL
--------------------------------------------------------
So, the #if branch is abandoned.

EDIT:

If I change the #if line to "#if A(2,3)", then I get two PASSes.
Code: [Select]
-PASS:  f234  f2345
-PASS:  f123  f1234
--------------------------------------------------------
Total 2 tests, 2 PASS, 0 FAIL
--------------------------------------------------------

« Last Edit: February 12, 2014, 09:33:06 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5247
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: The 08 February 2014 build (9639) is out.
« Reply #28 on: February 12, 2014, 02:48:24 pm »
@morten, revert rev9369 can avoid the hang issue, so I'm going to commit the change now. (This is a workaround, not a true fix).
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline osdt

  • Multiple posting newcomer
  • *
  • Posts: 61
Re: The 08 February 2014 build (9639) is out.
« Reply #29 on: February 12, 2014, 03:01:21 pm »
@morten, revert rev9369 can avoid the hang issue, so I'm going to commit the change now. (This is a workaround, not a true fix).

@ollydbg: reverting r9638 fixes this issue too!

I can confirm that r9639 has this issue while r9619 works flawlessly. Between this revisions, CC-related changes are r9636, r9638 and r9639.

- osdt
« Last Edit: February 12, 2014, 03:07:33 pm by osdt »