Codecompletion delays fixed in rev 1826 (update: in rev 1840)

[thomas]

Hmmm... I have slightly different figures.

Using this workspace:

Code

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_workspace_file>
	<Workspace title="All contrib plugins">
		<Project filename="cb\plugins\contrib\codestat\codestat.cbp" active="1" />
		<Project filename="cb\plugins\contrib\copystrings\copystrings.cbp" />
		<Project filename="cb\plugins\contrib\devpak_plugin\DevPakPlugin.cbp" />
		<Project filename="cb\plugins\contrib\help_plugin\help-plugin.cbp" />
		<Project filename="cb\plugins\contrib\keybinder\keybinder.cbp" />
		<Project filename="cb\plugins\contrib\profiler\cbprofiler.cbp" />
		<Project filename="cb\plugins\contrib\source_exporter\Exporter.cbp" />
		<Project filename="cb\plugins\contrib\wxSmith\wxSmith.cbp" />
		<Project filename="cb\CodeBlocks.cbp" />
	</Workspace>
</CodeBlocks_workspace_file>

I get 494.2 MB (with code completion) versus 12.6 MB (no code completion).

[attachment deleted by admin]

[thomas]

And here are the token counts.

Everything turned on:

Code

[18:12:07.281]: Start parsing project Code Stat
[18:12:07.281]: Concurrent threads for pool set to 1
[18:12:08.125]: Start parsing project copystrings
[18:12:08.125]: Concurrent threads for pool set to 1
[18:12:08.609]: Start parsing project Dev-C++ DevPak updater/installer (wx2.6)
[18:12:08.609]: Concurrent threads for pool set to 1
[18:12:09.765]: Start parsing project Help Plugin (wx2.6)
[18:12:09.765]: Concurrent threads for pool set to 1
[18:12:10.390]: Start parsing project C::B KeyBinder
[18:12:10.390]: Concurrent threads for pool set to 1
[18:12:11.109]: Start parsing project C::B Profiler
[18:12:11.109]: Concurrent threads for pool set to 1
[18:12:11.656]: Start parsing project Exporter
[18:12:11.656]: Concurrent threads for pool set to 1
[18:12:12.437]: Start parsing project wxSmith for wx 2.6
[18:12:12.437]: Concurrent threads for pool set to 1
[18:12:13.421]: Start parsing project Code::Blocks (wx2.6)
[18:12:13.421]: Concurrent threads for pool set to 1
[18:12:46.687]: Done parsing project Help Plugin (wx2.6) (550 total parsed files, 57254 tokens in 27.953 seconds).
[18:12:52.953]: Done parsing project C::B Profiler (514 total parsed files, 53061 tokens in 6.266 seconds).
[18:12:54.843]: Done parsing project Code Stat (506 total parsed files, 52749 tokens in 1.890 seconds).
[18:12:54.843]: Updating class browser...
[18:12:54.859]: Class browser updated.
[18:12:57.140]: Done parsing project Dev-C++ DevPak updater/installer (wx2.6) (536 total parsed files, 53977 tokens in 2.297 seconds).
[18:12:57.218]: Done parsing project Exporter (566 total parsed files, 57653 tokens in 0.78 seconds).
[18:12:58.531]: Done parsing project C::B KeyBinder (524 total parsed files, 55687 tokens in 1.313 seconds).
[18:13:00.031]: Done parsing project copystrings (573 total parsed files, 57413 tokens in 1.500 seconds).
[18:13:10.171]: Done parsing project wxSmith for wx 2.6 (841 total parsed files, 65030 tokens in 10.140 seconds).
[18:13:16.718]: Done parsing project Code::Blocks (wx2.6) (1210 total parsed files, 79395 tokens in 6.547 seconds).

Everything turned off:

Code

[18:15:16.468]: Start parsing project Code Stat
[18:15:16.468]: Concurrent threads for pool set to 1
[18:15:16.781]: Start parsing project copystrings
[18:15:16.781]: Concurrent threads for pool set to 1
[18:15:16.937]: Done parsing project Code Stat (7 total parsed files, 35 tokens in 0.0 seconds).
[18:15:16.937]: Updating class browser...
[18:15:16.937]: Class browser updated.
[18:15:17.078]: Start parsing project Dev-C++ DevPak updater/installer (wx2.6)
[18:15:17.078]: Concurrent threads for pool set to 1
[18:15:17.234]: Done parsing project copystrings (2 total parsed files, 9 tokens in 0.0 seconds).
[18:15:17.515]: Start parsing project Help Plugin (wx2.6)
[18:15:17.515]: Concurrent threads for pool set to 1
[18:15:17.687]: Done parsing project Dev-C++ DevPak updater/installer (wx2.6) (24 total parsed files, 348 tokens in 0.16 seconds).
[18:15:17.828]: Start parsing project C::B KeyBinder
[18:15:17.828]: Concurrent threads for pool set to 1
[18:15:18.062]: Done parsing project Help Plugin (wx2.6) (6 total parsed files, 59 tokens in 0.78 seconds).
[18:15:18.125]: Start parsing project C::B Profiler
[18:15:18.125]: Concurrent threads for pool set to 1
[18:15:18.296]: Done parsing project C::B KeyBinder (6 total parsed files, 396 tokens in 0.15 seconds).
[18:15:18.437]: Start parsing project Exporter
[18:15:18.437]: Concurrent threads for pool set to 1
[18:15:18.671]: Done parsing project C::B Profiler (6 total parsed files, 47 tokens in 0.78 seconds).
[18:15:18.875]: Start parsing project wxSmith for wx 2.6
[18:15:18.875]: Concurrent threads for pool set to 1
[18:15:19.062]: Done parsing project Exporter (21 total parsed files, 713 tokens in 0.78 seconds).
[18:15:19.234]: Start parsing project Code::Blocks (wx2.6)
[18:15:19.234]: Concurrent threads for pool set to 1
[18:15:19.656]: Done parsing project wxSmith for wx 2.6 (196 total parsed files, 3033 tokens in 0.266 seconds).
[18:15:24.312]: Done parsing project Code::Blocks (wx2.6) (547 total parsed files, 15636 tokens in 1.641 seconds).

[Ceniza]

When you say "Everything turned on" you mean "Parse preprocessor directives too?".

[thomas]

Yep. Everything.

[rickg22]

I think the whole model needs to be redesigned, it's obviously not fit for so many large projects at once...

OK assumming preprocessor directives take 5% of the total token count, and that the wxWidgets project gives us around 55,000 tokens... the sum of distinct tokens for all the workspace gives us a total of less than 90,000 tokens. And the parsing time would be less than 20 seconds.

Argh... I need to consult this with Yiannis.

[Ceniza]

Trying again with revision 1833 and everything on using Thomas' workspace.

[12:46:51.390]: Start parsing project Code::Blocks (wx2.6)
[12:46:51.390]: Concurrent threads for pool set to 2
[12:46:53.656]: Running startup script
[12:50:57.687]: Done parsing project copystrings (372 total parsed files, 45288 tokens in 244.0 seconds).
[12:51:00.234]: Done parsing project Code Stat (381 total parsed files, 45760 tokens in 2.547 seconds).
[12:51:00.234]: Updating class browser...
[12:51:00.265]: Class browser updated.
[12:51:11.421]: Done parsing project C::B KeyBinder (390 total parsed files, 46579 tokens in 11.187 seconds).
[12:51:12.484]: Done parsing project C::B Profiler (386 total parsed files, 45927 tokens in 1.63 seconds).
[12:51:30.593]: Done parsing project Help Plugin (wx2.6) (414 total parsed files, 47788 tokens in 18.109 seconds).
[12:51:31.578]: Done parsing project Exporter (423 total parsed files, 47872 tokens in 0.985 seconds).
[12:51:33.703]: Done parsing project Dev-C++ DevPak updater/installer (wx2.6) (441 total parsed files, 49220 tokens in 2.125 seconds).
[12:52:10.656]: Done parsing project wxSmith for wx 2.6 (644 total parsed files, 52045 tokens in 36.953 seconds).
[12:52:50.703]: Done parsing project Code::Blocks (wx2.6) (1083 total parsed files, 73764 tokens in 40.47 seconds).

It didn't crash this time!



Take a look at the CPU usage. Can you see what I meant?

[thomas]

And the parsing time would be less than 20 seconds.

Below 20 seconds?  :?
18:12:07 -- 18:13:16 = 69 seconds.
69 seconds * 0,95 = 65 seconds (more or less).

This is a fully optimized build, by the way (-DNDEBUG -O3 -fexpensive-optimizations -march=k8 -ffast-math). Debug build takes about twice as long.

[rickg22]

What I mean is that it would take 20 seconds if we decouple the tokens / files, so no file would be parsed more than once. This would imply that there would be one global parser, instead of one per project. It would also allow us to make a transition to a database if it was necessary.

[thomas]

Take a look at the CPU usage. Can you see what I meant?

Yes, and there is something else (besides the hyperthreading thing) to note. You can very clearly see that max_threads = NUM_CPUS is not a good choice.
Look at the regular "zero CPU" intervals, the CPU is stalled 50% of the time (very likely waiting for an I/O operation to finish). So, overall, you use only 25% of your CPU's capacity. It could very well do using two more threads, so one thread runs while another waits for I/O.

[rickg22]

So you recommend adding 1 or 2 extra threads?

Hmm interesting

[thomas]

Well, look at the screenshot. The CPU is completely idle 50% of the time (and runs one thread during the other half, when it could run two).

An even better solution might be to have exactly one dedicated I/O thread which does all reads asynchronously, but sequentially and feeds the results to the blocking NUM_CPU+1 worker threads.
Under Linux, this probably won't bring an awful lot, since the elevator mechanism should already make sure that the hard disk is not trashing when two threads try to read concurrently, but I wouldn't be so sure about that under Windows...

If someone can produce a cross-platform (posix + Windows) memory mapping class, then we might leave that work to the OS, too. It would probably be the most efficient way, anyway.

[Michael]

Just a suggestion. Would be of some interest the use of Win32 pthreads library in Windows? It was used to develop e.g., ViTooKi (a cross-platform C++ multimedia library).

Michael
 

[thomas]

Just a suggestion. Would be of some interest the use of Win32 pthreads library in Windows?

Win32 pthreads does not offer memory mapping, unluckily. The functionality which it offers is in wxWidgets too, so I don't think we can gain a lot from using it.

[Urxae]

If someone can produce a cross-platform (posix + Windows) memory mapping class, then we might leave that work to the OS, too. It would probably be the most efficient way, anyway.

Dare I suggest... Boost has such a class, in <boost/spirit/iterator/file_iterator.hpp>. It provides an iterator interface to files on disk.
There are basically 3 implementations (the appropriate one is selected using the preprocessor): One for Win32, one for Posix, and one using only the C++ standard library. The first two implement memory mapping, the other one is provided as a fallback.
If you don't want to include all of Boost though (which is understandable) you'll either have to check exactly which files are needed (and/or edit it to reduce dependencies manually) or perhaps just use it as an example for your own implementation.

The Boost license is GPL compatible, IIRC.

[Game_Ender]

The boost license is about a loose as it gets you just need to basically include the copyright from the author and that's about it.  Boost has a good thread library too.  If you are going to use an external thread library I would use boost, because then you can use lots of the other wonderful stuff boost has.

As for this I have one question, why are we using threads at all if doesn't run more than one at once?  I think you mean it only runs one <i>extra</i> thread, counting the main thread if you have one CPU?, or it only one thread total include the main one? The whole point of threads is to run more than the number of processors you have because the CPU is rarely full on, threads of execution spend quite a bit of time waiting.  Also I think the threading model needs a thorough rewrite or going over because it dead locks all the time on linux.