Print Page - NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => Topic started by: sodev on January 14, 2014, 11:00:59 pm

Title: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: sodev on January 14, 2014, 11:00:59 pm

When i open any of my projects that use the Visual Studio 2013 compiler or if they contain certain boost headers with the current version of CodeBlocks (r9573) CodeBlocks locks up when i open the first file. The debug window shows the following:

Code

NativeParser::OnParserStart(): Starting batch parsing for project 'visualizer'...
ReadVarNames() : Unexpected token '}' for 'plus', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xstddef', line 210.
ReadVarNames() : Unexpected token '}' for 'minus', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xstddef', line 225.
ReadVarNames() : Unexpected token '}' for 'multiplies', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xstddef', line 240.
ReadVarNames() : Unexpected token '}' for 'equal_to', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xstddef', line 255.
ReadVarNames() : Unexpected token '}' for 'less', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xstddef', line 270.
ReadVarNames() : Unexpected token '<' for '_Alloc_allocate', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xmemory0', line 313.
ReadVarNames() : Unexpected token '::' for 'shared_count', file 'C:/devel/boost_1_55_0/boost/smart_ptr/detail/shared_count.hpp', line 584.
ReadVarNames() : Unexpected token '}' for 'divides', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 211.
ReadVarNames() : Unexpected token '}' for 'modulus', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 224.
ReadVarNames() : Unexpected token '}' for 'negate', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 236.
ReadVarNames() : Unexpected token '}' for 'not_equal_to', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 252.
ReadVarNames() : Unexpected token '}' for 'greater', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 265.
ReadVarNames() : Unexpected token '}' for 'greater_equal', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 281.
ReadVarNames() : Unexpected token '}' for 'less_equal', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 294.
ReadVarNames() : Unexpected token '}' for 'logical_and', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 307.
ReadVarNames() : Unexpected token '}' for 'logical_or', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 320.
ReadVarNames() : Unexpected token '}' for 'logical_not', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 332.
ReadVarNames() : Unexpected token '}' for 'bit_and', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 345.
ReadVarNames() : Unexpected token '}' for 'bit_or', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 358.
ReadVarNames() : Unexpected token '}' for 'bit_xor', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 371.
ReadVarNames() : Unexpected token '}' for 'bit_not', file 'C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include/xfunctional', line 383.

After that CodeBlocks freezes. I checked the headers and these error messages occur when the parser sees a -> declytype() decoration of a template method, so i disabled the parsing of files without extension and i got a little further.

Code

NativeParser::OnParserStart(): Starting batch parsing for project 'visualizer'...
ReadVarNames() : Unexpected token '::' for 'shared_count', file 'C:/devel/boost_1_55_0/boost/smart_ptr/detail/shared_count.hpp', line 584.

And freeze again. Now this makes no sense because that line contains pretty ordinary c++ code and directly above pretty much the same line has passed without problems.

My last working CodeBlocks version is r9272, this version shows exactly the same error messages, however there is one more line:

Code

NativeParser::OnParserEnd(): Project 'visualizer' parsing stage done!

For some strange reason the very same errors don't produce a lockup. Every later revision i tested (only a few, can't remember which exactly) causes these lockups and is currently unusable for me. I tried to get the HandleMethod() function to at least recognize the decltype decoration but my code doesn't even get executed, i tried to understand the parser logic but after some hours i gave up.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: ollydbg on January 15, 2014, 03:26:36 pm

First, thanks for the report.

Generally, the messages like "ReadVarNames() : Unexpected token..." are debug log messages, they should not let C::B freeze, right?

When the Parser try to parse a project files, it firstly plot some message like

Code

NativeParser::OnParserStart(): Starting batch parsing for project 'visualizer'...

When it finishes paring the project files, it plot something like

Code

NativeParser::OnParserEnd(): Project 'visualizer' parsing stage done!

So, once you receive such message, this means the parsing job is done.

Quote

My last working CodeBlocks version is r9272

So, the bug may introduced between r9272 and r9573, but I think it is still hard to find which commit has regression.

Did you build C::B against wx2.8.12? Or wx3.0.0? I don't have VC in my system, but I can download the boost sources, is it possible to have a simple project that I can test it. Thanks.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: sigbjorn on January 15, 2014, 09:10:57 pm

I experience a similar lockup, arch linux x64 latest, codeblock 13.12, boost 1.55
Repeats: Open workspace with the projects. Parser starts up, no parsing stage done! message.

Update/workaround: Tweak with settings (Settings/Editor/Code completion C/C++ parser, turned off parse complex macros).
avoids hang, at the cost of less precise symbol parsing

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: oBFusCATed on January 15, 2014, 11:23:10 pm

Can someone extract a minimal sample file that reproduces the problem?

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: sodev on January 16, 2014, 09:10:46 pm

Hmm, apparently the problem must be something else, i created a minimal sample that just contained one file with a dummy int main() that includes the problematic boost header and although the error message appeared the parser completed this time. Even enabling the parsing of the VS system headers did not cause a lockup.

I'm trying to strip down my project to figure out what to remove to make the lockup disappear.

I am building CodeBlocks against wx 2.8.12 and the recent wxWidgets trunk and both versions show this behavior. Right now i am testing with the 2.8.12 version because CodeBlocks still causes assertions if using a higher wxWidgets version :(. I am compiling it with the latest TDM-MinGW32 4.8.1.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: sodev on January 18, 2014, 12:33:41 am

I found a way to easily produce the lockup, it doesn't even require Boost, however it does require to use Visual Studio as compiler for the project, i have used the 2013 version, don't know if it "works" also with older ones. All you need to do is to setup the include paths to include the Windows SDK that is shipped with the compiler, i haven't installed a full Windows SDK. Then open a project that contains this simple file:

Code

//-----------------------------------------------------------------------------
#include <windows.h>

int main() {
	return 0;
}

When CodeBlocks starts the parser it never finishes, i don't even get any error messages like before, CodeBlocks just freezes. This does only happen with Visual Studio, if TDM-MinGW is used as compiler it frankly produces some error messages but the parser finishes.

I am using default settings of CodeBlocks regarding the CodeCompletion plugin.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: stahta01 on January 18, 2014, 12:40:59 am

Quote from: sodev on January 18, 2014, 12:33:41 am

I found a way to easily produce the lockup, it doesn't even require Boost, however it does require to use Visual Studio as compiler for the project, i have used the 2013 version, don't know if it "works" also with older ones. All you need to do is to setup the include paths to include the Windows SDK that is shipped with the compiler, i haven't installed a full Windows SDK. Then open a project that contains this simple file:

Code
//-----------------------------------------------------------------------------
#include <windows.h>

int main() {
	return 0;
}
When CodeBlocks starts the parser it never finishes, i don't even get any error messages like before, CodeBlocks just freezes. This does only happen with Visual Studio, if TDM-MinGW is used as compiler it frankly produces some error messages but the parser finishes.

I am using default settings of CodeBlocks regarding the CodeCompletion plugin.

Can you post a list of CB Plugins that need to be on to cause the Lockup and what defines are done in the project.
I will see if I can dup with VS Express 2010.

Tim S.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: sodev on January 18, 2014, 12:56:23 am

I compiled CodeBlocks from SVN by using the CodeBlocks workspace and the ContribPlugins workspace and simply use whats in output folder after running the update script, i never bothered to disable any plugins. As for the project settings, this is my project file:

Code

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_project_file>
	<FileVersion major="1" minor="6" />
	<Project>
		<Option title="test" />
		<Option compiler="msvc10" />
		<Build>
			<Target title="Debug">
				<Option output="test_d.exe" prefix_auto="0" extension_auto="0" />
				<Option object_output="objvc/Debug" />
				<Option type="1" />
				<Option compiler="msvc10" />
				<Option use_console_runner="0" />
				<Compiler>
					<Add option="/Od" />
					<Add option="/Gm" />
					<Add option="/RTC1" />
					<Add option="/MTd" />
					<Add option="/Zi" />
					<Add option="/EHsc" />
					<Add option="/GR" />
					<Add option="/W4" />
					<Add option='/Fd&quot;objvc/Debug/&quot;' />
					<Add option="/DSTRICT" />
					<Add option="/D_CRT_SECURE_NO_DEPRECATE" />
					<Add option="/D_DEBUG" />
					<Add option="/DUNICODE" />
					<Add option="/D_UNICODE" />
					<Add option="/fp:precise" />
					<Add directory="src" />
				</Compiler>
				<Linker>
					<Add option="/DEBUG" />
					<Add option="/INCREMENTAL" />
					<Add option="/MANIFEST" />
					<Add option="/DYNAMICBASE" />
					<Add option="/NXCOMPAT" />
					<Add option='/MANIFESTUAC:&quot;level=&apos;asInvoker&apos; uiAccess=&apos;false&apos;&quot;' />
					<Add directory="." />
				</Linker>
			</Target>
			<Target title="Debug_DLL">
				<Option output="test_dll_d.exe" prefix_auto="0" extension_auto="0" />
				<Option object_output="objvc/Debug_DLL" />
				<Option type="1" />
				<Option compiler="msvc10" />
				<Option use_console_runner="0" />
				<Compiler>
					<Add option="/Od" />
					<Add option="/Gm" />
					<Add option="/RTC1" />
					<Add option="/MDd" />
					<Add option="/Zi" />
					<Add option="/EHsc" />
					<Add option="/GR" />
					<Add option="/W4" />
					<Add option='/Fd&quot;objvc/Debug_DLL/&quot;' />
					<Add option="/DSTRICT" />
					<Add option="/D_CRT_SECURE_NO_DEPRECATE" />
					<Add option="/D_DEBUG" />
					<Add option="/DUNICODE" />
					<Add option="/D_UNICODE" />
					<Add option="/fp:precise" />
					<Add directory="src" />
				</Compiler>
				<Linker>
					<Add option="/DEBUG" />
					<Add option="/INCREMENTAL" />
					<Add option="/MANIFEST" />
					<Add option="/DYNAMICBASE" />
					<Add option="/NXCOMPAT" />
					<Add option='/MANIFESTUAC:&quot;level=&apos;asInvoker&apos; uiAccess=&apos;false&apos;&quot;' />
					<Add directory="." />
				</Linker>
			</Target>
			<Target title="Release">
				<Option output="test.exe" prefix_auto="0" extension_auto="0" />
				<Option object_output="objvc/Release" />
				<Option type="1" />
				<Option compiler="msvc10" />
				<Option use_console_runner="0" />
				<Compiler>
					<Add option="/O2" />
					<Add option="/MT" />
					<Add option="/EHsc" />
					<Add option="/GR" />
					<Add option="/W4" />
					<Add option="/DSTRICT" />
					<Add option="/D_CRT_SECURE_NO_DEPRECATE" />
					<Add option="/DNDEBUG" />
					<Add option="/DUNICODE" />
					<Add option="/D_UNICODE" />
					<Add option="/fp:precise" />
					<Add directory="src" />
				</Compiler>
				<Linker>
					<Add option="/INCREMENTAL:NO" />
					<Add option="/MANIFEST" />
					<Add option="/DYNAMICBASE" />
					<Add option="/NXCOMPAT" />
					<Add option='/MANIFESTUAC:&quot;level=&apos;asInvoker&apos; uiAccess=&apos;false&apos;&quot;' />
					<Add directory="." />
				</Linker>
				<ExtraCommands>
					<Add after="mt.exe -manifest $exe_output.manifest -outputresource:$exe_output;1" />
					<Mode after="always" />
				</ExtraCommands>
			</Target>
			<Target title="Release_DLL">
				<Option output="test_dll.exe" prefix_auto="0" extension_auto="0" />
				<Option object_output="objvc/Release_DLL" />
				<Option type="1" />
				<Option compiler="msvc10" />
				<Option use_console_runner="0" />
				<Compiler>
					<Add option="/O2" />
					<Add option="/MD" />
					<Add option="/EHsc" />
					<Add option="/GR" />
					<Add option="/W4" />
					<Add option="/DSTRICT" />
					<Add option="/D_CRT_SECURE_NO_DEPRECATE" />
					<Add option="/DNDEBUG" />
					<Add option="/DUNICODE" />
					<Add option="/D_UNICODE" />
					<Add option="/fp:precise" />
					<Add directory="src" />
				</Compiler>
				<Linker>
					<Add option="/INCREMENTAL:NO" />
					<Add option="/MANIFEST" />
					<Add option="/DYNAMICBASE" />
					<Add option="/NXCOMPAT" />
					<Add option='/MANIFESTUAC:&quot;level=&apos;asInvoker&apos; uiAccess=&apos;false&apos;&quot;' />
					<Add directory="." />
				</Linker>
				<ExtraCommands>
					<Add after="mt.exe -manifest $exe_output.manifest -outputresource:$exe_output;1" />
					<Mode after="always" />
				</ExtraCommands>
			</Target>
		</Build>
		<Unit filename="src/main.cpp" />
		<Extensions>
			<code_completion />
			<envvars />
			<debugger />
		</Extensions>
	</Project>
</CodeBlocks_project_file>

Maybe i should note that im using the express version as well and, that doesn't apply to VS2010 though, that i use the Windows XP compatible SDK. The compiler variable in the project file reads msvc10 because CodeBlocks doesn't integrate higher versions, i just misuse that one and changed all the paths to point to 2013 :).

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: stahta01 on January 18, 2014, 01:16:07 am

I could NOT dup using VS Express 2010; but, I might try later using your project file.
No idea what else is needed to test.
I am out of Hard Disk space; so, download VS Expess 2013 is NOT something I plan to do.

Tim S.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: Alpha on January 18, 2014, 01:25:08 am

Quote from: sodev on January 18, 2014, 12:56:23 am

The compiler variable in the project file reads msvc10 because CodeBlocks doesn't integrate higher versions, i just misuse that one and changed all the paths to point to 2013 :).

Oops. Integration code has been sitting on my local version for a while now... I probably should tidy and commit it soon.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: sodev on January 20, 2014, 11:48:19 pm

So i've spend more time to try to solve this problem myself and got a little further, the cause of the lockup is a mutex deadlock because the parser thread gets stuck in an endless loop.

This is how i compiled CodeBlocks to track the error:

enabled CC_ENABLE_LOCKER_TRACK in cclogger.h
specified the defines CC_PROCESS_LOG_EVENT_TO_PARENT=1, CC_PARSERTHREAD_DEBUG_OUTPUT=1
fixed parserthread.cpp to not define CC_PARSERTHREAD_DEBUG_OUTPUT to zero and the beginning so that my compiler switches actually work
i disabled pretty much all TRACE calls in parserthread.cpp except the ones for found includes and tokenizer initialization

Opening my test project results in the following log: <log attached because of post size limit>

After it stoppped writing more debug messages i've clicked into the opened main.cpp and got one more line:

Code

s_TokenTreeMutex.Lock() : IsFileParsed(), C:\SVN\xternal\codeblocks\src\plugins\codecompletion\parser\parser.cpp, 1170

The GUI stopped responding after this, which makes sense because that mutex was still locked.

To track the problem further, i enabled more debug messages, but this ended up in chaos, i tried to enable file logging but that somehow didn't work, so all i have is a screenshot. However, these are the additional changes:

specified the additional defines CC_PARSER_DEBUG_OUTPUT=1, CC_TOKENIZER_DEBUG_OUTPUT=1
fixed parser.cpp and tokenizer.cpp to not override these defines

And this is the screenshot of the lockup: <see attachment>

So looks like that is the endless loop the thread is locked in, the last filename from the reduced log from the previous try was

Code

InitTokenizer() : m_Filename='C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Include\ObjBase.h', m_FileSize=38379.
Parse() : Parsing 'C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Include\ObjBase.h'

so it might be stuck in there or in some of its included headers. Now the strange thing is, if i include that file directly instead of windows.h it does not lockup?! The parsing finishes in that case!

[attachment deleted by admin]

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: ollydbg on January 21, 2014, 09:59:10 am

Hi, sodev, thanks for the testing.

Looking at your screen shot, I believe there is a bug in the Tokenizer/Parserthread, you see, all the log messages show Tokenizer is in line 863, and Current Char is E, and next Char is C. It is also the macro replacement functions which cause such bug, you can see ”_Group_impl_" is the key point. Maybe, some text replacement go into a endless loop, such as text "A" was replaced by text "B", then text "B" was replaced by text "C", then text "C" replaced by text "A" again.....

Here, Tokenizer dose not go ahead, so in this time, the Parserthread instance is holding the Tokentree, thus the mutex is never released. Now any access from the GUI thread (mostly in the event handler) to the Tokentree is blocked, so the main GUI get blocked too.

I would recommand to see what is the line 863 in the opened file, maybe parser are running endless loop in the file 'C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Include\ObjBase.h'.

Also, you can disable the macro-expansion feature in the CodeCompletion setting dialog, and see whether such loop happens again.

Since I can't reproduce your bug here(I don't have VC++ installed), so I can't give more help, sorry.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: sodev on January 21, 2014, 11:16:31 pm

The fragment you can see in the screenshot is part of the SAL macros from Visual Studio used for source code annotions, it can be found in sal.h

Code

// _When_(expr, annos) specifies that the annotations listed in 'annos' only
// apply when 'expr' evaluates to non-zero.
#define _When_(expr, annos)            _When_impl_(expr, annos _SAL_nop_impl_)
#define _Group_(annos)                 _Group_impl_(annos _SAL_nop_impl_)
#define _GrouP_(annos)                 _GrouP_impl_(annos _SAL_nop_impl_)

I couldn't find any of these macros in the SDK headers so it looks they are only used for the c runtime headers of Visual Studio, but _Group_impl_(annos _SAL_nop_impl_) doesn't appear anywhere outside of sal.h and even there it is only part of other defines. So that fragment in the debug message must be part of a deep nested define chain or something like that, can't figure out in what file the parser was when the deadlock happened.

Disabling parse complex macros in CodeBlocks made the deadlock go away, so for now this is my solution, i can't spend more time on messing around with CodeBlocks to tailor down the debug messages to figure out where it happens. It looks like SAL was introduced with Visual Studio 2012 so that explains why stahta01 couldn't reproduce with VS 2010.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: ollydbg on January 28, 2014, 04:15:48 pm

@sodev, can you try the latest nightly build version, when I fix the issue of Re: Code blocks using too much cpu (http://forums.codeblocks.org/index.php/topic,18824.msg129113.html#msg129113), I think your problem should also be solved, can you give some feedback?

Thanks.

Title: Re: NativeParser lockup when parsing Visual Studio 2013 and Boost 1.55.0 headers
Post by: sodev on January 29, 2014, 10:49:23 pm

I compiled r9614 and the lockup indeed is gone now, thanks for that.

Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => Topic started by: sodev on January 14, 2014, 11:00:59 pm