Author Topic: Hourglass in long operations (project loading)  (Read 30604 times)

Offline iw2nhl

  • Multiple posting newcomer
  • *
  • Posts: 116
  • BASIC, C, C++, Qt, bash
Hourglass in long operations (project loading)
« on: July 23, 2006, 04:23:41 pm »
When you open a big project (CodeBlocks.cbp is a good example), it takes some time to load all files.
C::B seems freezed during that time and nothing informs the user that it is working.
Could an hourglass mouse pointer be added during that time?
Another idea is a progress bar that shows the loading progress.
Obviously the 2 ideas can be used together ;-)

The same thing could be useful for other long operations (if any) except compilation because it already has it's feedback to the user (tool bar buttons that change color and build messages).

Offline Pecan

  • Plugin developer
  • Lives here!
  • ****
  • Posts: 2873
Re: Hourglass in long operations (project loading)
« Reply #1 on: July 23, 2006, 05:44:21 pm »
When you open a big project (CodeBlocks.cbp is a good example), it takes some time to load all files.
C::B seems freezed during that time and nothing informs the user that it is working.
Could an hourglass mouse pointer be added during that time?
Another idea is a progress bar that shows the loading progress.
Obviously the 2 ideas can be used together ;-)

The same thing could be useful for other long operations (if any) except compilation because it already has it's feedback to the user (tool bar buttons that change color and build messages).

I agree. I often re-click, then double click the codeblocks "recent projects"
html selection thinking I didn't hit it right. There's no indication that the user made a choice.

Offline Game_Ender

  • Lives here!
  • ****
  • Posts: 551
Re: Hourglass in long operations (project loading)
« Reply #2 on: July 23, 2006, 06:41:19 pm »

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1441
    • CenizaSOFT
Re: Hourglass in long operations (project loading)
« Reply #3 on: July 23, 2006, 07:03:53 pm »
It had a progress bar a long time ago but I don't know why it was removed. Adding the busy cursor sounds good, I just wonder who should add it or provide a patch :)

sethjackson

  • Guest
Re: Hourglass in long operations (project loading)
« Reply #4 on: July 23, 2006, 07:42:28 pm »
Busy cursor sounds good. :)

wxBusyCursor?
« Last Edit: July 23, 2006, 08:02:23 pm by sethjackson »

Offline iw2nhl

  • Multiple posting newcomer
  • *
  • Posts: 116
  • BASIC, C, C++, Qt, bash
Re: Hourglass in long operations (project loading)
« Reply #5 on: July 23, 2006, 08:10:39 pm »
The file opening seems to be done by the "OpenGeneric()" function.
It is called by:
void MainFrame::OnStartHereLink(wxCommandEvent& event)
bool MainFrame::OnDropFiles(wxCoord x, wxCoord y, const wxArrayString& files)
void MainFrame::OnFileReopenProject(wxCommandEvent& event)
void MainFrame::OnFileReopen(wxCommandEvent& event)
and the 5:
void MainFrame::OnFileImportProject[...](wxCommandEvent& event)

Analizing further more i noticed that "OnDropFiles()" calls it in a for() cicle so I think the hourglass should be put in these 2 functions:
OpenGeneric()
OnDropFiles()
This is to avoid OnDropFiles() showing a flashing hourglass because of the for() cicle.
With this function the hourglass will be asked twice, but it is not a problem because wxBeginBusyCursor() and wxEndBusyCursor() (thanks Game_Ender ;-)) are built for that use (they use a counter for nested calls).
No problems with wxBusyCursor() too (thanks sethjackson ;-)) because it calls the same 2 functions.

[EDIT]
From wxBusyCursor documentation:
just add
Code
wxBusyCursor wait;
as first lines of OpenGeneric() and OnDropFiles().
« Last Edit: July 23, 2006, 08:44:01 pm by iw2nhl »

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Hourglass in long operations (project loading)
« Reply #6 on: July 26, 2006, 09:33:07 am »
Project loading now takes almost exactly twice as long as it used to a couple of months back (for codeblocks.cbp, load time is around 1300 ms on my machine, which is just enough to be annoying). This is due to a several things which were added and which are not optimal. I talked about it with Yiannis a few weeks back.

A hourglass is generally not desired, instead the goal is to make project loading faster and partially asynchronous, so nothing like a hourglass (or progress bar) is needed.

The actual project loading takes only around 30-40 ms for codeblocks.cbp.
Some of the remaining time goes into parsing the project hierarchy (with a major part of CPU time going into the non-inlined cbC2U function and the wxString copy constructor). The major part, however, goes into checking for existing files, file write protection, and most notably, wxFileName overhead.
We haven't even started thinking about optimising the project loader, but that should be done (and will be done) eventually. Only not just now :)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline iw2nhl

  • Multiple posting newcomer
  • *
  • Posts: 116
  • BASIC, C, C++, Qt, bash
Re: Hourglass in long operations (project loading)
« Reply #7 on: July 26, 2006, 03:53:50 pm »
NOTE: this is only my opinion!

1)
I think that loading a project in an asyncronous way is not good because you cannot work on it reliably when it is partially loaded.
When I open codeblocks.cbp and I click on a menu, I find some menu-items disabled. This is not good because I think they are not available for a strange reason, while after some seconds thay become available.
I see this more like a bug than like a feature ;-).

2)
Loading time depends very much on computer speed, so you cannot say: it should be a fast task to execute.
On old PC it may take long time. Think also if loading a project from a slow device (USB, floppy, CD) or if C::B itself is on a slow device. Moreover there are people that use virtual OS (WMWare, QEMU, ...) to make tests and they are very slow.

3)
If you don't like my consideration above, consider at least this:
now loading is very slow also on fast PCs, so why not add a hourglass or something else until that code it optimized?

[EDIT]
- the disabled menu-items problem is visible also on little projects
- the first part of the loading for codeblocks.cbp freezes the program GUI (mouse shape is freezed too), this is another good reason for using an hourglass
« Last Edit: July 26, 2006, 04:10:45 pm by iw2nhl »

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Hourglass in long operations (project loading)
« Reply #8 on: July 26, 2006, 05:21:32 pm »
Quote
I think that loading a project in an asyncronous way is not good because you cannot work on it reliably when it is partially loaded.
This is not true as it is now, and it is not true for the things I was talking about making asynchronous in the future, either.
One of the things I was talking about that could easily run asynchronously was checking for file existence (or read-only flag). Code::Blocks loops through all project files and checks whether the corresponding file exists. If the file does not exist, a "broken file" icon is shown in the tree.
While this is a nice feature, it is generally not a good thing, as it is bound to the access times of a physical disk. We're talking about tens of milliseconds here, not nanoseconds.
Sure enough, there are file system caches, and they do work (otherwise loading the Code::Blocks project would take on the order of 10-15 seconds), but it is certainly not the best performer to run such a thing in the main thread while parsing the project structure.
There is nothing "unsafe" with running such queries asynchronously if proper synchronisation is done. If a file is really missing, you are shown an error if you try to actually open it, anyway. Thus the most "unsafe" thing to happen is displaying the wrong icon for half a second or so while the data is still being fetched. As project files typically do exist and are accessible, the chances for this to happen are quite odd.

Quote
When I open codeblocks.cbp and I click on a menu, I find some menu-items disabled. This is not good because I think they are not available for a strange reason, while after some seconds thay become available. I see this more like a bug than like a feature ;-).
This bug is what actually makes it reliable. You can only do certain things after the project was fully loaded.

Quote
Loading time depends very much on computer speed, so you cannot say: it should be a fast task to execute. On old PC it may take long time.
I did not say so, and yes indeed, on a slower PC, it will take longer. That's obvious, but that's not the point.
There has been a lot of research in the design of user interfaces, and people are generally more willing to accept a 10 second "busy time" if they are still able to do something than a 500 ms delay if you lock them out entirely (for example by showing a progress bar window or a "busy" cursor).

What we are therefore trying to do (in order of significance) is:
1. make it work
2. make it non-modal, non-blocking, as far as possible
3. make it fast

Quote
now loading is very slow also on fast PCs
This is not quite true. On a reasonably "normal" machine (3 GHz, no SMP, no dual-core, no Hyperthreading), project loading takes on the order of 100-200 ms for a small project and on the order of 1.2-1.4 seconds for a project the size of Code::Blocks. Most IDEs are significantly slower than that. Even though we have not yet spent one minute optimising the project loader, we really don't need to hide under a rock.

The thing that really takes a very long time to get running is the code completion plugin (up to tens of seconds). For some reason which I don't remember, it blocks parts of the toolbar and menus, too. However, this has nothing to do with project loading per se.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline iw2nhl

  • Multiple posting newcomer
  • *
  • Posts: 116
  • BASIC, C, C++, Qt, bash
Re: Hourglass in long operations (project loading)
« Reply #9 on: July 26, 2006, 05:34:30 pm »
Then we could divide the loading in 2 parts: syncronous and asyncronous.
While the syncronous part is being executed, I think there should be an hourglass (or other). Then the user can start working while the asyncronous part is being done.

Sorry, but a 3 GHz CPU is not so "normal"!

When code completion parsing starts?
Probably the long time I see (about 10 seconds, 1.8 GHz CPU) is related to that.

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Hourglass in long operations (project loading)
« Reply #10 on: July 26, 2006, 07:55:36 pm »
Quote
Then we could divide the loading in 2 parts: syncronous and asyncronous.
While the syncronous part is being executed, I think there should be an hourglass (or other). Then the user can start working while the asyncronous part is being done.
That's what we're partially doing already, and that's what we will probably extend in the future, as described above.
Showing a hourglass should not be necessary any more once we have had a thorough optimisation pass. Taking into account that we query around 5,000 attributes (which we could fold to around 100-200), needlessly construct-convert-copy-construct around 4,000 wxString objects, and re-calculate the common toplevel path around 200 times while loading codeblocks.cbp (and call wxFileName::Normalize similarly often), there is plenty of opportunity to make the synchronous part faster. Some of these optimisations will not be trivial, though, and some would require changing the project file format.

I shall not commit myself to a precise number, but I think it is not entirely unreasonable to expect reducing the current load time to maybe 25% (or less?) of the current time if putting some serious work into that.
If all the other things that do not necessarily need to run synchronously are moved out of the main line, then you should not have much of a noticeable delay left, so a hourglass should really not be needed at all.

However, it is not optimisation time yet (mandrav dixit). To date, we're happy if we get it to work. Once it works, we can make it faster at a later time.

Quote
Sorry, but a 3 GHz CPU is not so "normal"!
Yeah well... a lot of people run 4 GHz and upwards CPUs these days, many of them with either hyperthreading or dual-core (or both). So actually, a plain normal 3 GHz CPU is comparatively slow :)

Don't get me wrong, though. A 4 GHz CPU, 2 GB of physical RAM, and 900 MB of hard disk storage are not and will never be a system requirement to get Code::Blocks running :lol:

Quote
When code completion parsing starts?
Immediately after the actual project loading has finished, but before the GUI comes "alive" again.
Quote
Probably the long time I see is related to that.
Yes, very likely.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline iw2nhl

  • Multiple posting newcomer
  • *
  • Posts: 116
  • BASIC, C, C++, Qt, bash
Re: Hourglass in long operations (project loading)
« Reply #11 on: July 26, 2006, 09:34:27 pm »
Ok.

Seen that for this version the optimizations won't be done and that there is no user feedback that something is occurring, while not show an hourglass for now? It can be removed anytime in the future. It is not a big work, just 1 line of code!
I tried to add it and recompile: the effect is very good. Moreover the program seems to me more responsive because it reacts immediately to the click in the initial HTML page.
(See for this also the 2nd message by Pecan)

About that page, it is not updated at the right time: it should go away just after the click, not when all the project has been loaded.

[EDIT]
Another thing: this problem is not just for "project" loading, but also for files loading.
Try to drag-and-drop 20 or more source files in C::B, it takes a long time and the GUI is freezed.
That single line of code solves this problem too.
« Last Edit: July 26, 2006, 10:03:00 pm by iw2nhl »

Offline killerbot

  • Administrator
  • Lives here!
  • *****
  • Posts: 5529
Re: Hourglass in long operations (project loading)
« Reply #12 on: July 26, 2006, 10:17:47 pm »
I have tried this, I think the hourglass is not that bad, it's good.

However it seems (I stress "seems") I have some crashes on closing Cb (with my cb workspace (the codeblocks.cbp and all contrib cbp's)à. All of them , off course, in the code completion dll. So I don't think the wxBusyCursor is bad, just that darn codecompletion.dll. Since a change 2 months ago it became to me a big pain in the ...

One thing I want to mention about the file loading, 2 weeks ago, I did the same thing opened up 20-30 files at the same time. It ws awfull :
 - slow
 - CB was like a X-mass tree, those menus flickering ....

We should solve this rather soon then later, and that codecompletion, that it is does not always work and it has it's limitations, but at this moment it's a crash chance of above 50 %  :-(

[EDIT] : a few more tries , and no crash now

for everyone's info ;-)
Code
Error occured on Wednesday, July 26, 2006 at 22:11:07.

E:\data\killerbot\CodeBlocks\svn\src\devel\codeblocks.exe caused an Access Violation at location 65ec2863 in module E:\data\killerbot\CodeBlocks\svn\src\devel\share\codeblocks\plugins\codecompletion.dll Reading from location 00690054.

Registers:
eax=08ee8f10 ebx=0a35fcc0 ecx=0069004c edx=0069004c esi=ffffffff edi=00000000
eip=65ec2863 esp=0a35fc24 ebp=0a35fc24 iopl=0         nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

Call stack:
65EC2863  E:\data\killerbot\CodeBlocks\svn\src\devel\share\codeblocks\plugins\codecompletion.dll:65EC2863  PluginSDKVersion
65F01A9B  E:\data\killerbot\CodeBlocks\svn\src\devel\share\codeblocks\plugins\codecompletion.dll:65F01A9B  std::_Rb_tree_const_iterator<int>::operator++()  C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../include/c++/3.4.5/bits/stl_tree.h:255
65E8F081  E:\data\killerbot\CodeBlocks\svn\src\devel\share\codeblocks\plugins\codecompletion.dll:65E8F081  ClassBrowserBuilderThread::AddTreeNode(wxTreeItemId const&, Token*, bool)  E:/data/killerbot/CodeBlocks/svn/src/plugins/codecompletion/classbrowserbuilderthread.cpp:276
65E8EC3D  E:\data\killerbot\CodeBlocks\svn\src\devel\share\codeblocks\plugins\codecompletion.dll:65E8EC3D  ClassBrowserBuilderThread::AddTreeNamespace(wxTreeItemId const&, Token*, std::set<unsigned, std::less<unsigned>, std::allocator<unsigned> > const&)  E:/data/killerbot/CodeBlocks/svn/src/plugins/codecompletion/classbrowserbuilderthread.cpp:191
65E8E000  E:\data\killerbot\CodeBlocks\svn\src\devel\share\codeblocks\plugins\codecompletion.dll:65E8E000  ClassBrowserBuilderThread::BuildTree()  E:/data/killerbot/CodeBlocks/svn/src/plugins/codecompletion/classbrowserbuilderthread.cpp:103
65E8D867  E:\data\killerbot\CodeBlocks\svn\src\devel\share\codeblocks\plugins\codecompletion.dll:65E8D867  ClassBrowserBuilderThread::Entry()  E:/data/killerbot/CodeBlocks/svn/src/plugins/codecompletion/classbrowserbuilderthread.cpp:30
61A609D5  E:\data\killerbot\CodeBlocks\svn\src\devel\wxmsw26u_gcc_cb.dll:61A609D5  _ZN8wxThreadD2Ev
77BCB530  C:\WINDOWS\syswow64\msvcrt.dll:77BCB530  _endthreadex
7D4E0729  C:\WINDOWS\syswow64\kernel32.dll:7D4E0729  FlsSetValue
« Last Edit: July 26, 2006, 10:24:43 pm by killerbot »

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Hourglass in long operations (project loading)
« Reply #13 on: July 27, 2006, 01:20:29 am »
Quote
Another thing: this problem is not just for "project" loading, but also for files loading.
Try to drag-and-drop 20 or more source files in C::B, it takes a long time and the GUI is freezed.
That single line of code solves this problem too.
It does not solve any problem. All it does is show a different cursor. That's following the wrong path. The right path would be to make it behave so it doesn't take that long at all and doesn't freeze.
Besides, the GUI is perfectly responsive under Linux while opening 50 files via D&D (not that it opens them any faster, though). The freeze seems to be a Windows thing.
I have an idea what the reason for D&D being so slow might be, too, but hopefully I am wrong...

Quote
CB was like a X-mass tree, those menus flickering
Much of the flickering when opening/closing many files comes from not disabling the editor notebooks. The code was commented out for some reason a long time ago (don't know why). I uncommented it in cf a week or two ago to test whether there are any problems and there seem to be none, so we can probably leave that. It'll eventually get merged.

Another problem is the updating of menus and toolbars in general, but that won't be an easy thing to solve. We are currently doing everything via update_ui events, which is not optimal at all. We are sending way too many events and do way too many things over and over again needlessly.
However, I would not touch that part of the IDE now since Yiannis has implemented his action-based menu system. Tampering with menus now would mean to do useless work, as you would basically have to write a kind of menu manager (which is what we already have).
While it is quite inefficient, the current implementation has one advantage: it works.

Quote
codecompletion, that it is does not always work and it has it's limitations, but at this moment it's a crash chance of above 50 %
I don't think we'll ever be able to fix this plugin, nobody understands what is happening anyway. For now (until the new one is production quality), I have code completion entirely disabled, and everything works smoothly. I never see a freeze or a crash (except for the one and only known wxSmith crash-on-exit issue) or any other bad condition.
It's just a shame you don't have "jump to implementation" that way. But then, that never works correctly anyway.
Post tenebras lux... :)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline iw2nhl

  • Multiple posting newcomer
  • *
  • Posts: 116
  • BASIC, C, C++, Qt, bash
Re: Hourglass in long operations (project loading)
« Reply #14 on: July 27, 2006, 01:37:15 am »
It does not solve any problem. All it does is show a different cursor. That's following the wrong path. The right path would be to make it behave so it doesn't take that long at all and doesn't freeze.
Besides, the GUI is perfectly responsive under Linux while opening 50 files via D&D (not that it opens them any faster, though). The freeze seems to be a Windows thing.
What I mean is that now the code is slow and takes a long time.
When the new code will be ready, we will speak about that way of working.
Now we have another code, a slow code.