Author Topic: Symbols browser issue of CC has been fixed for my Linux  (Read 2482 times)

Offline Miguel Gimenez

  • Regular
  • ***
  • Posts: 383
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #15 on: November 26, 2019, 08:14:17 pm »
The second (dump)


Edit: both are executed in the main thread, and the first number was mistakenly including the second. In my next message the numbers are corrected and compared with the new version
« Last Edit: November 27, 2019, 04:30:52 pm by Miguel Gimenez »

Offline Miguel Gimenez

  • Regular
  • ***
  • Posts: 383
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #16 on: November 27, 2019, 12:31:51 pm »
I have changed the std::list to a kind of intrusive list (each element has pointers to parent, previous and next siblings and first child).

I still have to adapt the quicksort algorithm, but the timings with CB project look better. Below are the old times (tree creation times revised) wth the new ones at the right side (remember, no sort).

- Current projects symbols
    - tree creation 36 ms, now 8 ms
    - top tree dump 227 ms, now 15 ms
    - bottom tree (creation+dump)
        - Global functions 40+132, now 21+105
        - Global typedefs 16+29, now 13+26
        - Global variables 32+97, now 17+77
        - Macro definitions 390+1093, now 167+707

- Everything
    - tree creation 427 ms, now 241 ms
    - top tree dump 452 ms, now 159 ms
    - bottom tree (creation+dump)
        - Global functions 1720+7702, now 838+2264
        - Global typedefs 634+2003, now 329+1496
        - Global variables 183+526, now 114+435
        - Macro definitions 12321+26790, now 5803+14174

The top tree creation is made in the worker thread, while the dump and the bottom tree work are made in the main thread.

Offline Miguel Gimenez

  • Regular
  • ***
  • Posts: 383
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #17 on: November 28, 2019, 11:23:01 am »
Quicksort reactivated; the timings for tree creation have increased, but not much

21 -> 30
13 -> 14
17 -> 23
167 -> 215
838 -> 978
329 -> 384
114 -> 143
5803 -> 6211

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12129
    • Travis build status
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #18 on: November 30, 2019, 10:33:20 am »
The posts you're making aren't very clear.
As I've said earlier we're interested only/mostly in the timings in the main thread.
How long does it take to execute the workers is not that important (it is, but it is secondary).
The question is what is taking the time? You'll have to use a profiler. Try either vtune or perf.
VTune is easier to setup for stack-gathering and it recently switched to be free instead of being mightily expensive.

As I've said previously the main thread must not pause for more than 10ms during this operation.
After you know what is slow you could decide what is best course of action. I won't be surprised that the tree-ctrl api is slow.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5247
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #19 on: November 30, 2019, 10:41:22 am »
One idea: why not creating the whole tree, is there any optimized way we can build some part of the tree which are shown to the user. That's only a very small part of the whole tree. If user scroll up and down the tree, we have to build the tree gradually.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12129
    • Travis build status
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #20 on: November 30, 2019, 10:43:44 am »
For lists this concept/mode is called virtual list. We use this to make opening the alt-g dialog to be really fast no matter the size of the project (the dialog opens in <100ms even on the linux kernel project (40+k files)).

I'm not sure this is available for trees though :( Probably we could do create-children-on-expand thing, but I'm not sure if this would benefit us.

As I've said we first need to know what is slow then we could find a solution. And more importantly why it is slow. Then we'll be able to find a solution. Guessing about performance is never a good idea.
« Last Edit: November 30, 2019, 10:45:48 am by oBFusCATed »
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Miguel Gimenez

  • Regular
  • ***
  • Posts: 383
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #21 on: November 30, 2019, 11:59:10 am »
I was using wxStopWatch.

The code works the same as before, only the dump parts have been added, and they use the main thread because it is the only one allowed to access the CCTreeCtrl.

- The top tree creation is made in the worker thread as before.
- The worker thread only activates (now and before) when the top tree changes due to a code completion token tree change.
- The top tree dump is made by the main thread when the tree is finished (this part is new).
- The bottom tree creation and dump are made by the main thread in the OnSelect event (just like before). Of course, the dump adds delay.

The tree leaves are created dynamically when expanding their parent, and deleted when the parent is collapsed, just like in the original code. This work is (and was) made by the main thread.

I find it quite usable if you don't use the Everything option.

I have attached my final changes, just in case anybody wants to test/profile/modify or reuse parts.

Offline earlgrey

  • Multiple posting newcomer
  • *
  • Posts: 102
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #22 on: November 30, 2019, 09:00:20 pm »
Seems to work, thank you.
* OS = Debian Buster - Linux 4.19.06 x64 SMP
* C::B = svn11267 wx-3.0.4 - Linux, unicode 64 bit

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5247
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #23 on: December 02, 2019, 03:57:36 pm »
I was using wxStopWatch.

The code works the same as before, only the dump parts have been added, and they use the main thread because it is the only one allowed to access the CCTreeCtrl.

- The top tree creation is made in the worker thread as before.
- The worker thread only activates (now and before) when the top tree changes due to a code completion token tree change.
- The top tree dump is made by the main thread when the tree is finished (this part is new).
- The bottom tree creation and dump are made by the main thread in the OnSelect event (just like before). Of course, the dump adds delay.

The tree leaves are created dynamically when expanding their parent, and deleted when the parent is collapsed, just like in the original code. This work is (and was) made by the main thread.

I find it quite usable if you don't use the Everything option.

I have attached my final changes, just in case anybody wants to test/profile/modify or reuse parts.
I try to apply the patch file "symbol_browser.patch" in my local git repo.

But it looks like I have no way to apply the patch. :(

Code: [Select]
$ patch -p0 ../symbol_browser.patch

This will make the patch command hangs without  returning, I have to press the "CTRL+C" to halt the patch command. (I'm using the shell from git-for-windows)


EDIT:
There is another issue, I see this patch contains some changes like:
Code: [Select]
-            const TokenIdxSet* tokens = tree->GetTokensBelongToFile(*itf);
+            const TokenIdxSet *tokens = tree->GetTokensBelongToFile(*itf);
I would still prefer the old styling.  :)
« Last Edit: December 02, 2019, 04:01:06 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Miguel Gimenez

  • Regular
  • ***
  • Posts: 383
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #24 on: December 02, 2019, 06:40:03 pm »
Here you have a revised copy with the coding standard restored. I prefer the other way, but obviously the existing standard must be respected.

The patch is generated with svn (svn diff > symbol_browser.patch), I can generate it with svn diff --git if you need it.
in the next message you have a version generated with --git option.

I have been testing timings in Symbol Browser with 17.12, and the delays in main thread are similar (as I said above, most of the work in the bottom tree is, and was, made by the main thread).
« Last Edit: December 02, 2019, 06:47:25 pm by Miguel Gimenez »

Offline Miguel Gimenez

  • Regular
  • ***
  • Posts: 383
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #25 on: December 02, 2019, 06:45:56 pm »
I have attached a patch file generated with svn diff --git

Offline kipade

  • Multiple posting newcomer
  • *
  • Posts: 44
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #26 on: December 10, 2019, 04:04:19 am »
I have attached a patch file generated with svn diff --git
after several busy days, I tried this patch and it works find for me.
I use git, use this command to do:
Code: [Select]
git apply ~/Downloads/symbol_browser_git.patch  -p2
Thanks for this perfect work and patch.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12129
    • Travis build status
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #27 on: December 10, 2019, 09:24:41 am »
Thanks for this perfect work and patch.
I guess your project is either small or you could tolerate long UI pauses.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Miguel Gimenez

  • Regular
  • ***
  • Posts: 383
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #28 on: December 10, 2019, 02:03:10 pm »
Most of the pauses are already in the original code, the greatest one is in the OnSelect() event due to full bottom tree creation in main thread.

Offline kipade

  • Multiple posting newcomer
  • *
  • Posts: 44
Re: Symbols browser issue of CC has been fixed for my Linux
« Reply #29 on: December 11, 2019, 02:24:25 am »
Most of the pauses are already in the original code, the greatest one is in the OnSelect() event due to full bottom tree creation in main thread.
Yes, you are right. there are some short pauses when click the top tree items.
Since it have to be done in the main UI thread, I think no way to avoid that if did not keep all parsed symbols tree item in memory.