Author Topic: potential switcher questions  (Read 9579 times)

Offline taukimus

  • Single posting newcomer
  • *
  • Posts: 9
potential switcher questions
« on: January 31, 2013, 04:10:37 am »
I am attempting to switch from Source Insight to Code::Blocks (I would like to save the company some money).

I have tried a few things, and I'm not sure I'm setting things up correctly, but I've noticed a few things that are making the switch difficult to sell to others here.  I shall list them here.  First, let me describe my configuration.

0. C::B 12.11 full install
1. Dell Precision T3600 8G. Windows 7 Enterprise.  Code::Blocks is run in Admin mode. Virus/malware scanner has an exception for anything in C::B directory.
2. All code is located on a network filer, accessed using SMB mapped drives.  This one is not changeable, unfortunately.
3. I have created one giant project with all files in it.  The reason for this is that we have a number of platforms we support, with some shared files and many not shared.  I am also often creating and destroying views (we use ClearCase) so the codebase I work on isn't the same for very long. Creating a project per set of shared files for each platform would be the correct way to do it, but it is cumbersome, and is wasted time if the view is going away in a few days.  Setting up some scripts to autogenerate the project structure might work out since the project structure is basically the same, just the files included are different.  I would still need to use the "1 parser per workspace" mode I guess because I need all the symbols from all the projects in the code completion.
4. The project consists of 10451 files with 16.7M sloc (used the code statistics plugin for that!).
5. The project files will never be inside the same tree as the source files.  There are very few developers here who like having their source code mixed in with its meta data (docs, diagrams, project files, etc. -- Makefiles are the only exception).
6. Code is a mix of C/C++ files, Makefiles, and perl.  The perl runs under x86 (on a build server) to generate some code.  All other code is cross-compiled (on a build server) for some embedded targets.
7. Most of the files are not changing every day.
8. Due to confidentiality, I cannot submit any of my projects or source as examples.  I'm sorry.

And now for the comments.  Please do not take these as complaints, but merely as observations and comparisons.  You all have done great work so far. =)

A. Workspace Management

A.1. Project load time

It takes roughly a minute to load this project in C::B, during which Windows believes C::B is not responding.  Loading the same project in SI takes 2 seconds.  SI doesn't have the ability to tell me in the tree control that a file is RW/RO, nor does it know if the file exists or not until I try to open it.  I read some discussions here regarding all the things you do at project load time (I was trying to track down why it was so slow for me) to build the tree and I wonder if some of them are not really necessary or should have options to disable.  For example, updating whether a file is there or not or RW/RO when I try to open it might be good enough (rather than at project open time).  Also, I'm not sure, but it looks like SI is doing a lazy load of the tree structure as I browse through it (rather than all at once).  That might help here too (are you creating the entire tree structure at project open time?).  If the tree nodes themselves can be cached/reused that might help (with speed and mem usage) if you aren't doing that already (or wxWidgets isn't).  Also, SI makes me specify the top level root directory of my project at project creation time.  This keeps them from having to discover this, which I understand can be a source of a lot of the load time (and some interesting logic) in C::B (and some improvements are on the way it seems).  Any files in SI which reside under the project path are stored with relative paths, all others are absolute.

A.2. Open last workspace on startup

I can't find any option to open the last workspace I was using (including all window positions and open projects and files) on startup.  This would be really helpful.  The only option I found was default or blank, neither of which reopens projects I had open when I shutdown the app last time.  If this exists, please let me know where.

A.2. List view

I would love to see a list view of all the project files with a combo box with search capability like SI has.  This is not the same thing as disabling the "Display folders as on disk" option.  I use the SI list control exclusively rather than browse a file tree looking for stuff, as more often than not, I know exactly what file I'm looking for, and its just plain faster to start typing it in, get it autocompleted, hit enter, and its open!  I realize that the tree is better for project exploration, and I realize that the list with search is a feature request (does wxWidgets even provide something like this?).  Where do I submit feature requests?  Bugtracker?

A.3. Context window

SI has a really cool reference tracking thingie that is really useful.  Basically, it is just a window pane that displays the location of the code to which your cursor is referring.  So, for example, if your cursor is on a function invocation, it displays the definition of that function in the context window.  It does this on the fly.  References which resolve to multiple entries show up as a list so you can pick between them.  Clicking on the code in the context window opens that file and goes to that line in a main editor window.  This is another feature request unless there is a plugin which does this already and I've missed it?

B. Code completion

B.1. Parse time.

It takes overnight to parse this project (I let it go for 4 hours yesterday, and it was still going when I left work.  Today when I came in, it was finally done, so I'm not exactly sure how long it took).  During this entire parse time, the editor interface is laggy (just clicking to position the cursor takes some time, and typing is behind by 1/2 char to a few chars) and at random times the entire interface becomes completely unresponsive (according to windows) for a minute or so. CPU utilization is only 5-7% the entire time. Memory was up to about 1G, # handles > 10k during the parsing. Disabling the code completion plugin makes the lagginess and hangs go away, but, well, no code completion then.  Disabling the code completion toolbar seems to have no effect on the lagginess.

For comparison, SI creates its cc database for the first parse within 5 minutes for the same project if I ask it to do it in the foreground (get a progress bar and everything).  The SI background parser takes much longer (about an hour), but doesn't interfere with the editing of any files (it also updates the status window when not in use with the paths of the files it is parsing).  If references haven't been parsed by the time you try to look one up, then it just says so.  But really, the up-front foreground parsing with incremental background re-parsing is really fast.  Perhaps another feature request?

[Update] I did another full reparse today and it only took a little over an hour.  That's good, and matches SI's background parse.  However, I can't use the editor during this time.  If I could get a consistent 1 hour bg parse and not have it interfere with the editor, and if I could get an option to foreground parse that is much shorter...

B.2. Reparse every time?

It looks like I have to wait to reparse each project every time I close/reopen C::B.  SI saves the parse data in a database file (not sure what they use).  This means that I don't have to reparse each time I load up a project, which takes tons of time (see B.1).  Is there an option to save the parse and disable the reparse on each load?  Does your parser check for modified files (since last reparse time) and only reparse those?  Could it? =)

C. Symbol browser

C.1. Pane sizing

It seems this thing resizes itself to maximize the bottom pane and minimize the top one, especially when parsing is incomplete.  I think it would be better if it resized the top pane to fit the contents, and gave whatever is left to the bottom window.  Or allow a user to position it and save that setting in the workspace, maybe?  Or both (with an option to choose between auto-resize and user-fixed size)?

C.2. Symbol parsing

It looks like the symbol parsing for the current focused file editor is not done until the entire project parse is done (or maybe it's because the file I'm testing is way down in the list).  SI always does the currently-focused file right away, as it is usually a quick parse to do one file.  If symbol references to other files are not finished yet in the overall bg parse, then they show up as undefined / not linked.

D. Editor window

I look forward to the renewed efforts (Alpha) in Semantic Highlighting going on in the fora.  Based on comments, I can see some issues with the way code completion, symbol browsing, and SH are currently coupled, though.

E. Building/compiling

I would love to be building over ssh using custom Makefiles, with the build results parsed by C::B (and remote debugging would be awesome, too).  I saw this: http://forums.codeblocks.org/index.php/topic,6289.0.html, but there were a few issues that weren't resolved in that thread.  And that was 2007; is there a doc somewhere that describes how to do this in the newer versions?

Thanks for listening.  I really like this software and I want to make it better (and I would love to convice all the SIers at work to use it instead).  I may be able to spend some time working on some of the features or bugs I've mentioned.
« Last Edit: January 31, 2013, 04:13:43 am by taukimus »

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5915
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: potential switcher questions
« Reply #1 on: January 31, 2013, 04:58:53 am »
Firstly, welcome to C::B forum, and thanks for the time to write the comments.
Quote
B. Code completion

B.1. Parse time.

It takes overnight to parse this project (I let it go for 4 hours yesterday, and it was still going when I left work.  Today when I came in, it was finally done, so I'm not exactly sure how long it took). 
You can start Codeblocks with the option: "--debug-log" in command line argument, then, in the "Codeblocks Debug panel", there is a message telling you how long the parser runs, like:
Quote
Project 'Code::Blocks wx2.8.x' parsing stage done (1438 total parsed files, 128497 tokens in 0 minute(s), 22.094 seconds).


Quote
During this entire parse time, the editor interface is laggy (just clicking to position the cursor takes some time, and typing is behind by 1/2 char to a few chars) and at random times the entire interface becomes completely unresponsive (according to windows) for a minute or so. CPU utilization is only 5-7% the entire time. Memory was up to about 1G, # handles > 10k during the parsing. Disabling the code completion plugin makes the lagginess and hangs go away, but, well, no code completion then.  Disabling the code completion toolbar seems to have no effect on the lagginess.
The parsing is done in NON-GUI thread, so I think the lagginess is a bug, but how to reproduce this issue? We need a way to reproduce this issue.

Quote
For comparison, SI creates its cc database for the first parse within 5 minutes for the same project if I ask it to do it in the foreground (get a progress bar and everything).  The SI background parser takes much longer (about an hour), but doesn't interfere with the editing of any files (it also updates the status window when not in use with the paths of the files it is parsing).  If references haven't been parsed by the time you try to look one up, then it just says so.  But really, the up-front foreground parsing with incremental background re-parsing is really fast.  Perhaps another feature request?

[Update] I did another full reparse today and it only took a little over an hour.  That's good, and matches SI's background parse.  However, I can't use the editor during this time.  If I could get a consistent 1 hour bg parse and not have it interfere with the editor, and if I could get an option to foreground parse that is much shorter...

B.2. Reparse every time?

It looks like I have to wait to reparse each project every time I close/reopen C::B.  SI saves the parse data in a database file (not sure what they use).  This means that I don't have to reparse each time I load up a project, which takes tons of time (see B.1).  Is there an option to save the parse and disable the reparse on each load?  Does your parser check for modified files (since last reparse time) and only reparse those?  Could it? =)
Codeblocks' CodeCompletion plugin currently do not save parser's database to harddisk, so all the database was created in memory when a new project/workspace loaded. Some of this feature implemented like saving tokens to harddisk and loading them back, but there are a lot of other things not implemented. Mostly the database are big, like 200M+ or even bigger in your case, I think it should still take a lot of time when handling this database.

Any way, C::B is a free/open source software, so any contributions are welcome. Thanks.

If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline taukimus

  • Single posting newcomer
  • *
  • Posts: 9
Re: potential switcher questions
« Reply #2 on: January 31, 2013, 06:06:49 am »
Quote
Firstly, welcome to C::B forum, and thanks for the time to write the comments.

Thank you.

Quote
You can start Codeblocks with the option: "--debug-log" in command line argument, then, in the "Codeblocks Debug panel", there is a message telling you how long the parser runs, like:
Quote
Project 'Code::Blocks wx2.8.x' parsing stage done (1438 total parsed files, 128497 tokens in 0 minute(s), 22.094 seconds).

Aha! I was using a -d option which I gleaned from somewhere in here.  Most likely outdated.  I'm trying --debug-log now.

Quote
The parsing is done in NON-GUI thread, so I think the lagginess is a bug, but how to reproduce this issue? We need a way to reproduce this issue.

Can we use some other huge open source project (how big are apache or xorg or linux kernel?) over a windows share on a local machine? That is, share the folder the project is in, mount it as a network drive locally (and hope that windows isn't smart enough to know it's really a local drive). Also, I was thinking (dangerous, I know).  I wonder if even though the parser and editor are on different threads, maybe they are both accessing something that is synchronized (maybe way down deep in wx), and as such it causes these hiccups?  Also, the lag always exists for me while the parser is active, but it is spiking/pulsing from very small lag all the way up to just past 1 minute of "not responding" with the hourglass/spiral.  It seems almost periodic or perhaps associated with each new file that is parsed...

Quote
Codeblocks' CodeCompletion plugin currently do not save parser's database to harddisk, so all the database was created in memory when a new project/workspace loaded. Some of this feature implemented like saving tokens to harddisk and loading them back, but there are a lot of other things not implemented. Mostly the database are big, like 200M+ or even bigger in your case, I think it should still take a lot of time when handling this database.

SI's databases just for this project total 355M (sym db: 163M; sym ref idx: 19M; main sym idx: 22M; aux sym idx: 146M; prj file: 3M [C::B is only 1.5M]; some other stuff).  They use indexes of the main db for speed, which they save to disk (I don't think they are fully in memory while the project is open, otherwise SI would have a >58M commit footprint).  Definitely a space/speed tradeoff there.  Probably should be an option.

Quote
Any way, C::B is a free/open source software, so any contributions are welcome. Thanks.

I will try. =)

Offline taukimus

  • Single posting newcomer
  • *
  • Posts: 9
Re: potential switcher questions
« Reply #3 on: January 31, 2013, 06:09:18 am »
Or maybe even the lagginess is caused by hitting some concurrent connection limit to the share?  Does the editor constantly do things with the file (looking for modifications perhaps)?

Offline taukimus

  • Single posting newcomer
  • *
  • Posts: 9
Re: potential switcher questions
« Reply #4 on: January 31, 2013, 06:10:26 am »
I have "check for externally modified files" unchecked, and it is still laggy.

Offline taukimus

  • Single posting newcomer
  • *
  • Posts: 9
Re: potential switcher questions
« Reply #5 on: January 31, 2013, 11:06:59 pm »
Quote
Project 'nexus-97231-latest' parsing stage done (10140 total parsed files, 464089 tokens in 70 minute(s), 12.445 seconds).

70 mins sounds good for a background parse.  I don't know what happened the first time I tried it and it took longer than 4 hours...


Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: potential switcher questions
« Reply #6 on: January 31, 2013, 11:47:18 pm »
Quote
A.2. Open last workspace on startup
Missing, you can fake it if you save your workspace as the default one. It might be easy feature to add.

Quote
A.2. List view
Press Alt+G or use Search->Goto File

Quote
A.3. Context window
Can you show us a video showing this feature in action, it sounds interesting.

Quote
Where do I submit feature requests?  Bugtracker?
Look at codeblocks.org there are some useful links at the side.

About the slow opening of it would be good if you can provide a profile by some program like Intel VTune.

Quote
2. All code is located on a network filer, accessed using SMB mapped drives.  This one is not changeable, unfortunately.
Why aren't you using VCS? This is never going to be fast or safe.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline taukimus

  • Single posting newcomer
  • *
  • Posts: 9
Re: potential switcher questions
« Reply #7 on: February 01, 2013, 12:29:09 am »
Thanks for the responses.  I've just tried Alt-G.  Exactly what I'm looking for (although it flickers while typing the file name, which is a bit annoying).

I'll see if I can capture a video of the context window in action.  Basically, imagine that whenever you would use the "jump to decl/implementation" feature of C::B, you also have a window that shows the context / code of where your jump target is.  So you don't have to "jump to decl/implementation" if you just need a quick browse of a struct or function.  I suppose it would be like having a split in your source window where one pane always has your current editor, and the other one always has the location of "jump to decl/impl" displayed (and it tracks when you move the caret).

Does VCS mean something other than a Version Control System here?  We are using IBM Rational Clearcase.  The main Clearcase database is stored on (a) redundant server(s) with backup (I'm sure IT/CM team have this part properly safeguarded).  All of our developer Clearcase views are stored on a filer NFS-mounted by our build server and our targets.  We do not build locally because we need everyone to have exactly the same build env / toolchains (and we need NFS access to run the binaries on each target), and we have many platforms to support, so we have 20 odd toolchains active at the same time.  This is a lot for each developer to get right on their own machine and created an IT headache last time we tried it (they are also big on IP and network security, so only certain machines can expose NFS, TFTP, etc.).  For example, I personally have 60 sets of code ("views" in Clearcase) for different releases and platforms that I need to examine, bug-fix, add features, all of which have 10k or more files.  So I need to switch projects a few times per day, and the load and reparse is killing me.  I would agree that this would naturally not be as quick as having the views on my local hard drive, but since SI has no speed problems with the network views, I can't honestly say that being on the network causes speed problems in general.
« Last Edit: February 01, 2013, 12:55:21 am by taukimus »

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: potential switcher questions
« Reply #8 on: February 01, 2013, 01:15:31 am »
I can't honestly say that being on the network causes speed problems in general.
In fact it does, everything is 10-20 times slower, so there is a need for more careful programming.
In order to find the cause for the slow loading we will need a profile or a test case. A fake one probably will be enough.
Have you tried to disable CC?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline taukimus

  • Single posting newcomer
  • *
  • Posts: 9
Re: potential switcher questions
« Reply #9 on: February 01, 2013, 01:59:43 am »
Allow me to adjust my comments: yes, running over the network causes things to run slower.  However, what I mean is that if SI can still do things quickly while having the files all located over SMB, then there must be a way to do it quickly in C::B.

Disabling CC doesn't improve the load time.

Quote
10451 files loaded
Done loading project in 54900ms

However, as I noted previously, it removes the UI lag entirely.

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: potential switcher questions
« Reply #10 on: February 01, 2013, 05:53:52 am »
Re project load speed, I think the patch discussed here is in trunk and just needs to be enabled with the appropriate #define when building CB. I think that should improve load speed quite a bit! (There could be related issues with path handling in the CC plugin that need similar patching, or it could be something else entirely)

Offline taukimus

  • Single posting newcomer
  • *
  • Posts: 9
Re: potential switcher questions
« Reply #11 on: February 02, 2013, 02:02:21 am »
Link to SI's context window feature demo video.

http://s1292.beta.photobucket.com/user/taukimus/media/context_zpsb4143cc2.mp4.html

You will need to download the video to see it clearly, I think.  Also, VLC didn't capture the mouse position, so to see what is going on you'll need to watch the caret position and see how moving it changes the context window at the bottom.

Offline taukimus

  • Single posting newcomer
  • *
  • Posts: 9
Re: potential switcher questions
« Reply #12 on: February 02, 2013, 02:03:09 am »
Thanks dmoore.  I'll see if I can get a build system up and running to make latest trunk.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5915
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: potential switcher questions
« Reply #13 on: February 20, 2013, 03:35:25 pm »
Link to SI's context window feature demo video.

http://s1292.beta.photobucket.com/user/taukimus/media/context_zpsb4143cc2.mp4.html

You will need to download the video to see it clearly, I think.  Also, VLC didn't capture the mouse position, so to see what is going on you'll need to watch the caret position and see how moving it changes the context window at the bottom.
I just watch the video, do you mean when the caret position changes, the docked window at the bottom automatically show the source snippet of the type declaration?

In C::B, you can right click on a symbol, then select the "find declaration of xxxxx", after that, you will go to its position as you shown in the "context window".

I think its a feature request, right?
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline jarod42

  • Multiple posting newcomer
  • *
  • Posts: 87
Re: potential switcher questions
« Reply #14 on: February 20, 2013, 06:26:05 pm »
In VS, this window is called 'Code definition Window'.
(but VS do (random?) choose in case of ambiguity :/ )

The window is automatically updated when caret position change.
It is a permanent show declaration of xxxx.