There must be something wrong with it。
When I open an Project.
There are lost a word like like this "NativeParser::OnParserEnd(): Project '******' parsing stage done!".
and I can do nothing with it,it FC.
Where do you see this messages ?
Can you provide a sample (minimal) workspace/project where this occurs ?
build(9619):
The message is like this:
NativeParser::CreateParser(): Finish creating a new parser for project '***********'
NativeParser::OnParserEnd(): Project '********' parsing stage done!
build(9639):
like this:
NativeParser::CreateParser(): Finish creating a new parser for project '***********'
Only one word.
Ok, so it looks like the CC's parser hangs or go to some infinite loop.
If you see the change between rev 9619 and 9639, there are few CC related changed.
Revision: 9639
Author: ollydbg
Date: 2014-2-8 17:33:14
Message:
- CC: use wx_str() instead of c_str().
-------------------------------
M(T ) : /trunk/src/plugins/codecompletion/nativeparser.cpp
Revision: 9638
Author: ollydbg
Date: 2014-2-8 17:32:33
Message:
* CC: fix a bug (logic error) introduced in rev 9601.
-------------------------------
M(T ) : /trunk/src/plugins/codecompletion/parser/tokenizer.cpp
Revision: 9636
Author: fuscated
Date: 2014-2-4 7:15:50
Message:
- CC: Make the code a bit more readable
-------------------------------
M(T ) : /trunk/src/plugins/codecompletion/parser/parserthread.cpp
Revision: 9635
Author: fuscated
Date: 2014-2-4 7:15:45
Message:
- wx30: Fix a format specifier mismatch assert
-------------------------------
M(T ) : /trunk/src/plugins/codecompletion/parser/parserthread.cpp
I guess the change of rev9638 (* CC: fix a bug (logic error) introduced in rev 9601.) cause this bug. But in-fact rev 9638 is going fix a logic error, so another guess is that rev 9601 does not truly fix the bug.
I would welcome a test project/workspace. Meanwhile, I will re-test the sample project in this discussion: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers (http://forums.codeblocks.org/index.php/topic,18801.msg128775.html#msg128775) later today.
EDIT: Code blocks using too much cpu (http://forums.codeblocks.org/index.php/topic,18824.msg128966.html#msg128966), this contains a sample project which will lead to hang issue.
What I tried is to create a new wxWidgets project, without PCH, a debug configuration (which uses the wxWidgets 3.0.0 release libs) and a release configuration. The project contains no code changes, only the generated files. After opening the project, the CPU load increases up to 100% and CB can only be killed via Task Manager.
Here is the content of the project file (I don't want to provide any zip archives):
?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_project_file>
<FileVersion major="1" minor="6" />
<Project>
<Option title="minimal_test" />
<Option pch_mode="2" />
<Option compiler="gcc" />
<Build>
<Target title="Debug">
<Option output="bin/minimal_test" prefix_auto="1" extension_auto="1" />
<Option object_output="bin/" />
<Option type="0" />
<Option compiler="gcc" />
<Option projectLinkerOptionsRelation="2" />
<Compiler>
<Add option="-g" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</Compiler>
<ResourceCompiler>
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</ResourceCompiler>
<Linker>
<Add library="libwxmsw30u_core.a" />
<Add library="libwxbase30u.a" />
<Add library="libwxpng.a" />
<Add library="libwxzlib.a" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib" />
</Linker>
</Target>
<Target title="Release">
<Option output="bin/minimal_test" prefix_auto="1" extension_auto="1" />
<Option object_output="bin/" />
<Option type="0" />
<Option compiler="gcc" />
<Option projectLinkerOptionsRelation="2" />
<Compiler>
<Add option="-O2" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</Compiler>
<ResourceCompiler>
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</ResourceCompiler>
<Linker>
<Add option="-s" />
<Add library="libwxmsw30u_core.a" />
<Add library="libwxbase30u.a" />
<Add library="libwxpng.a" />
<Add library="libwxzlib.a" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib" />
</Linker>
</Target>
</Build>
<Compiler>
<Add option="-pipe" />
<Add option="-mthreads" />
<Add option="-D__GNUWIN32__" />
<Add option="-D__WXMSW__" />
<Add option="-DwxUSE_UNICODE" />
<Add option="-Wall" />
<Add directory="C:/Libs/wxWidgets_gcc/include" />
</Compiler>
<ResourceCompiler>
<Add directory="C:/Libs/wxWidgets_gcc/include" />
</ResourceCompiler>
<Linker>
<Add option="-mthreads" />
<Add library="libkernel32.a" />
<Add library="libuser32.a" />
<Add library="libgdi32.a" />
<Add library="libwinspool.a" />
<Add library="libcomdlg32.a" />
<Add library="libadvapi32.a" />
<Add library="libshell32.a" />
<Add library="libole32.a" />
<Add library="liboleaut32.a" />
<Add library="libuuid.a" />
<Add library="libcomctl32.a" />
<Add library="libwsock32.a" />
<Add library="libodbc32.a" />
</Linker>
<Unit filename="minimal_testApp.cpp" />
<Unit filename="minimal_testApp.h" />
<Unit filename="minimal_testMain.cpp" />
<Unit filename="minimal_testMain.h" />
<Unit filename="resource.rc">
<Option compilerVar="WINDRES" />
</Unit>
<Extensions>
<code_completion />
<envvars />
<debugger />
<lib_finder disable_auto="1" />
</Extensions>
</Project>
</CodeBlocks_project_file>
What I tried is to create a new wxWidgets project, without PCH, a debug configuration (which uses the wxWidgets 3.0.0 release libs) and a release configuration. The project contains no code changes, only the generated files. After opening the project, the CPU load increases up to 100% and CB can only be killed via Task Manager.
Here is the content of the project file (I don't want to provide any zip archives):
?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_project_file>
<FileVersion major="1" minor="6" />
<Project>
<Option title="minimal_test" />
<Option pch_mode="2" />
<Option compiler="gcc" />
<Build>
<Target title="Debug">
<Option output="bin/minimal_test" prefix_auto="1" extension_auto="1" />
<Option object_output="bin/" />
<Option type="0" />
<Option compiler="gcc" />
<Option projectLinkerOptionsRelation="2" />
<Compiler>
<Add option="-g" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</Compiler>
<ResourceCompiler>
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</ResourceCompiler>
<Linker>
<Add library="libwxmsw30u_core.a" />
<Add library="libwxbase30u.a" />
<Add library="libwxpng.a" />
<Add library="libwxzlib.a" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib" />
</Linker>
</Target>
<Target title="Release">
<Option output="bin/minimal_test" prefix_auto="1" extension_auto="1" />
<Option object_output="bin/" />
<Option type="0" />
<Option compiler="gcc" />
<Option projectLinkerOptionsRelation="2" />
<Compiler>
<Add option="-O2" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</Compiler>
<ResourceCompiler>
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib/mswu" />
</ResourceCompiler>
<Linker>
<Add option="-s" />
<Add library="libwxmsw30u_core.a" />
<Add library="libwxbase30u.a" />
<Add library="libwxpng.a" />
<Add library="libwxzlib.a" />
<Add directory="C:/Libs/wxWidgets_gcc/lib/gcc_lib" />
</Linker>
</Target>
</Build>
<Compiler>
<Add option="-pipe" />
<Add option="-mthreads" />
<Add option="-D__GNUWIN32__" />
<Add option="-D__WXMSW__" />
<Add option="-DwxUSE_UNICODE" />
<Add option="-Wall" />
<Add directory="C:/Libs/wxWidgets_gcc/include" />
</Compiler>
<ResourceCompiler>
<Add directory="C:/Libs/wxWidgets_gcc/include" />
</ResourceCompiler>
<Linker>
<Add option="-mthreads" />
<Add library="libkernel32.a" />
<Add library="libuser32.a" />
<Add library="libgdi32.a" />
<Add library="libwinspool.a" />
<Add library="libcomdlg32.a" />
<Add library="libadvapi32.a" />
<Add library="libshell32.a" />
<Add library="libole32.a" />
<Add library="liboleaut32.a" />
<Add library="libuuid.a" />
<Add library="libcomctl32.a" />
<Add library="libwsock32.a" />
<Add library="libodbc32.a" />
</Linker>
<Unit filename="minimal_testApp.cpp" />
<Unit filename="minimal_testApp.h" />
<Unit filename="minimal_testMain.cpp" />
<Unit filename="minimal_testMain.h" />
<Unit filename="resource.rc">
<Option compilerVar="WINDRES" />
</Unit>
<Extensions>
<code_completion />
<envvars />
<debugger />
<lib_finder disable_auto="1" />
</Extensions>
</Project>
</CodeBlocks_project_file>
OK, many thanks. I can reproduce the hang issue with the latest C::B SVN head, so I will try to see what cause this hang issue.
Here is the BT that I see m_TokenIndex always stay in the same value (infinite loop)
[debug]#0 Tokenizer::ReplaceMacro (this=0x940fc98, str="wxDEFINE_UNICHARREF_OPERATOR") at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\tokenizer.cpp:1259
[debug]#1 0x65ef64b1 in Tokenizer::DoGetToken (this=0x940fc98) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\tokenizer.cpp:1230
[debug]#2 0x65ef5e19 in Tokenizer::PeekToken (this=0x940fc98) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\tokenizer.cpp:1064
[debug]#3 0x65ee3ba9 in ParserThread::DoParse (this=0x940fc90) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.cpp:1002
[debug]#4 0x65ee7107 in ParserThread::HandleClass (this=0x940fc90, ct=ParserThread::ctClass) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.cpp:1971
[debug]#5 0x65ee3262 in ParserThread::DoParse (this=0x940fc90) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.cpp:811
[debug]#6 0x65ee23df in ParserThread::Parse (this=0x940fc90) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.cpp:513
[debug]#7 0x65f0607e in ParserThread::Execute (this=0x940fc90) at F:\cb_sf_git\trunk\src\plugins\codecompletion\parser\parserthread.h:155
[debug]#8 0x010ab44b in cbThreadPool::cbWorkerThread::Entry (this=0x6bab438) at F:\cb_sf_git\trunk\src\sdk\cbthreadpool.cpp:216
[debug]#9 0x62764ba9 in wxThreadInternal::DoThreadStart(wxThread*) () from E:\code\wx-mingw-build-481-dw2\wxWidgets-2.8.12\lib\gcc_dll\wxmsw28u_gcc_custom.dll
[debug]#10 0x62764c85 in wxThreadInternal::WinThreadStart(void*)@4 () from E:\code\wx-mingw-build-481-dw2\wxWidgets-2.8.12\lib\gcc_dll\wxmsw28u_gcc_custom.dll
[debug]#11 0x77c3a3b0 in msvcrt!_endthreadex () from C:\WINDOWS\system32\msvcrt.dll
[debug]#12 0x7c80b729 in KERNEL32!GetModuleFileNameA () from C:\WINDOWS\system32\kernel32.dll
[debug]#13 0x00000000 in ?? ()
Here is the BT that I see m_TokenIndex always stay in the same value (infinite loop)
For me its different: It looks like in bool ParserThread::Parse() in the else if (!switchHandled) clause it always end in the m_Str << token << ParserConsts::space_chr; line which will crash sooner or later if the wxString cannot append more characters. at that time, token is something like (wxMACRO(...)) which cannot be resolved / handled in this loop.
I debugged a little, I think it was a regression after a patch by Huki, that is:
Revision: f7cb07c8dcf613f6786eaf0ee34e908ec218d2c8
Author: ollydbg <ollydbg@2a5c6006-c6dd-42ca-98ab-0921f2732cef>
Date: 2013-9-29 16:25:10
Message:
* CC: reliable working of UngetToken() when macro expansion is involved, it set the undo index value correctly(thanks Huki)
* CC: avoid take backward step (UngetToken) twice in the Tokenizer class
- CC: comments added for Tokenizer class
git-svn-id: https://svn.code.sf.net/p/codeblocks/code/trunk@9369 2a5c6006-c6dd-42ca-98ab-0921f2732cef
----
Modified: src/plugins/codecompletion/parser/tokenizer.cpp
Modified: src/plugins/codecompletion/parser/tokenizer.h
which have such diff:
@@ -1148,6 +1148,8 @@ wxString Tokenizer::PeekToken()
unsigned int savedLineNumber = m_LineNumber;
unsigned int savedNestLevel = m_NestLevel;
+ int savedReplaceCount = m_IsReplaceParsing ? m_RepeatReplaceCount : -1;
+
if (SkipUnwanted())
m_PeekToken = DoGetToken();
else
@@ -1156,17 +1158,33 @@ wxString Tokenizer::PeekToken()
m_PeekTokenIndex = m_TokenIndex;
m_PeekLineNumber = m_LineNumber;
m_PeekNestLevel = m_NestLevel;
-
- m_TokenIndex = savedTokenIndex;
- m_LineNumber = savedLineNumber;
- m_NestLevel = savedNestLevel;
+ // Check whether a ReplaceBufferForReparse() was done in DoGetToken().
+ // We assume m_Undo... have already been reset in ReplaceBufferForReparse().
+ if (m_IsReplaceParsing && savedReplaceCount != (int)m_RepeatReplaceCount)
+ {
+ m_TokenIndex = m_UndoTokenIndex;
+ m_LineNumber = m_UndoLineNumber;
+ m_NestLevel = m_UndoNestLevel;
+ }
+ else
+ {
+ m_TokenIndex = savedTokenIndex;
+ m_LineNumber = savedLineNumber;
+ m_NestLevel = savedNestLevel;
+ }
}
return m_PeekToken;
So, bug is here:
wxString Tokenizer::PeekToken()
{
if (!m_PeekAvailable)
{
m_PeekAvailable = true;
unsigned int savedTokenIndex = m_TokenIndex;
unsigned int savedLineNumber = m_LineNumber;
unsigned int savedNestLevel = m_NestLevel;
int savedReplaceCount = m_IsReplaceParsing ? m_RepeatReplaceCount : -1;
if (SkipUnwanted())
m_PeekToken = DoGetToken();
else
m_PeekToken.Clear();
m_PeekTokenIndex = m_TokenIndex;
m_PeekLineNumber = m_LineNumber;
m_PeekNestLevel = m_NestLevel;
// Check whether a ReplaceBufferText() was done in DoGetToken().
// We assume m_Undo... have already been reset in ReplaceBufferText().
if (m_IsReplaceParsing && savedReplaceCount != (int)m_RepeatReplaceCount)
{
m_TokenIndex = m_UndoTokenIndex; //*************bug here******************
m_LineNumber = m_UndoLineNumber;
m_NestLevel = m_UndoNestLevel;
}
else
{
m_TokenIndex = savedTokenIndex;
m_LineNumber = savedLineNumber;
m_NestLevel = savedNestLevel;
}
}
return m_PeekToken;
}
m_TokenIndex is always remain/reset to the same value, so Tokenizer won't go forward any more.
The code do the macro replacement is a mass, the big error looks like it is not the way a C Processor do the replacement.
For example, if we have such code
/****************************************/
#define A(x,y) (x+y)
#define B 1
#define C 2
#if A(B,C)
void f1234();
#endif
void f2345();
// f123 //f1234
// f234 //f2345
But from my point of view, our Tokenizer don't have a "rescan" stage, the trick thing is that it did some text substitute on formal parameter to actual parameter translation, some kind of recursive call.
Note, with the latest cc-test frame, I get such result
-PASS: f234 f2345
*FAIL: f123 f1234
--------------------------------------------------------
Total 2 tests, 1 PASS, 1 FAIL
--------------------------------------------------------
EDIT:
If I change the #if line to "#if A(2,3)", then I get two PASSes.
-PASS: f234 f2345
-PASS: f123 f1234
--------------------------------------------------------
Total 2 tests, 2 PASS, 0 FAIL
--------------------------------------------------------
@morten, revert rev9369 can avoid the hang issue, so I'm going to commit the change now. (This is a workaround, not a true fix).
@ollydbg: reverting r9638 fixes this issue too!
I can confirm that r9639 has this issue while r9619 works flawlessly. Between this revisions, CC-related changes are r9636 and r9638.
- osdt
Dear osdt, thanks for the reply. Rev9638 is a solid fix for rev 9601, because when in rev 9601, I leave a logic error.
See what rev9638 does:
src/plugins/codecompletion/parser/tokenizer.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/plugins/codecompletion/parser/tokenizer.cpp b/src/plugins/codecompletion/parser/tokenizer.cpp
index 6761574..25b0a6d 100644
--- a/src/plugins/codecompletion/parser/tokenizer.cpp
+++ b/src/plugins/codecompletion/parser/tokenizer.cpp
@@ -1985,7 +1985,7 @@ bool Tokenizer::GetMacroExpendedText(const Token* tk, wxString& expandedText)
// sanity check if we have such macro definition that #define AAA(x,y) x+y+AAA
// which means a macro name is exists in its definition, which will cause a infinite expansion loop
- if (tk->m_FullType.Find(tk->m_Name) ==wxNOT_FOUND)
+ if (tk->m_FullType.Find(tk->m_Name) != wxNOT_FOUND)
return false;
// Now, tk is a function like macro definition we are going to expand, it's m_Args contains the
I debugged a little, I think it was a regression after a patch by Huki,
So, bug is here:
// Check whether a ReplaceBufferText() was done in DoGetToken().
// We assume m_Undo... have already been reset in ReplaceBufferText().
if (m_IsReplaceParsing && savedReplaceCount != (int)m_RepeatReplaceCount)
{
m_TokenIndex = m_UndoTokenIndex; //*************bug here******************
m_LineNumber = m_UndoLineNumber;
m_NestLevel = m_UndoNestLevel;
}
Are we dealing with really complex nested macros in wx3.0? I'm assuming that is the problem...
A quick explanation of my patch for PeekToken(): the goal is to reset m_TokenIndex every time a macro replacement occurs inside ReplaceBufferText(). So we use the test "if (m_IsReplaceParsing && savedReplaceCount != (int)m_RepeatReplaceCount)", to see if m_RepeatReplaceCount has increased after calling DoGetToken(). This should be safe because there is a limit on the number of nested macro replacements allowed (s_MaxRepeatReplaceCount), so infinite loop is impossible.
But seeing the concerned code, I think the problem is obvious: (in tokenizer.cpp)
bool Tokenizer::ReplaceBufferText(const wxString& target, bool updatePeekToken)
{
if (target.IsEmpty())
return false;
if (m_IsReplaceParsing && ++m_RepeatReplaceCount > s_MaxRepeatReplaceCount)
{
m_TokenIndex = m_BufferLen - m_FirstRemainingLength;
m_PeekAvailable = false;
SkipToEOL(false);
return false;
}
...
We can see that the m_RepeatReplaceCount will keep increasing even after s_MaxRepeatReplaceCount is reached, even though no more replacement occurs. We should change it to something like this:
bool Tokenizer::ReplaceBufferText(const wxString& target, bool updatePeekToken)
{
if (target.IsEmpty())
return false;
if (m_IsReplaceParsing && m_RepeatReplaceCount >= s_MaxRepeatReplaceCount)
return false;
++m_RepeatReplaceCount;
...
@ollydbg: Does my patch still cause the hang with the above fix?
...
@ollydbg: Does my patch still cause the hang with the above fix?
@Huki, many thanks, fairly clear explanation, I just test the code and no hang now.
This is the code I use:
bool Tokenizer::ReplaceBufferText(const wxString& target, bool updatePeekToken)
{
if (target.IsEmpty())
return false;
if (m_IsReplaceParsing)
{
if (m_RepeatReplaceCount >= s_MaxRepeatReplaceCount)
{
m_TokenIndex = m_BufferLen - m_FirstRemainingLength;
m_PeekAvailable = false;
SkipToEOL(false);
return false;
}
else
++m_RepeatReplaceCount;
}
.....
BTW: I originally thought the test condition in PeekToken() can be changed to
if (m_IsReplaceParsing && savedReplaceCount < (int)m_RepeatReplaceCount)
Because in some cases, m_RepeatReplaceCount will reset to zero, but finally I found that is not necessary, because both m_IsReplaceParsing and m_RepeatReplaceCount will be reset to zero in the same time.
if (m_FirstRemainingLength != 0 && m_BufferLen - m_FirstRemainingLength < m_TokenIndex)
{
m_FirstRemainingLength = 0;
m_IsReplaceParsing = false;
m_RepeatReplaceCount = 0;
}
BTW2: Why we need two variables? I think m_IsReplaceParsing is a redundant variable of m_RepeatReplaceCount. Checking all the source code, I realize that
RepeatReplaceCount > 0 means m_IsReplaceParsing == true
RepeatReplaceCount == 0 means m_IsReplaceParsing == false
if (savedReplaceCount < m_RepeatReplaceCount)
But the code to initialize savedReplaceCount can be simply
size_t savedReplaceCount = m_RepeatReplaceCount;
Then
can change to
if (m_RepeatReplaceCount > 0)
Also need to adjust the code below
// Set replace parsing state, and save first replace token index
if (!m_IsReplaceParsing)
{
m_FirstRemainingLength = m_BufferLen - m_TokenIndex;
m_IsReplaceParsing = true;
}
I will prepare two commits:
1, fix the hang issue as you suggest.
2, code refactoring by remove the m_IsReplaceParsing variable.
Finally, thanks for your help!