Semantic highlight

[daniloz]

Hi guys!
I'm almost done with the very basic real-time semantic highlighter (first parsing is quite slow, that is, 4s but next ones highlight sources quite fast, i.e., below 0.5s on my Pentium Dual-Core machine). The plugin crashes sometimes so I'm not going to attach the sources, but I'm working on it
...

I'm looking forward to try it...

[perento]

Its still very earlier to ask but this just pop in my mind and wondered what do you think.
So..what you think on highlighting a member variables and is it possible? Keep the work, please.

[mistar]

Its still very earlier to ask but this just pop in my mind and wondered what do you think.
So..what you think on highlighting a member variables and is it possible? Keep the work, please.

It will be possible to highlight differently all different (semantic) kinds of variables, i.e., global, local, member (public/private/static/non-static/...), parameter variables, and similarly for functions.
I have little time now, so my work on SH will be delayed a little bit.

[mistar]

Simple question: how to check whether C::B startup is completed? Is there any function for this?
I know there is EVT_APP_STARTUP_DONE event but I need to check whether I am after this event.
For example when I load my plugin via Plugins->Manage plugins, C::B is ready in my OnAttach, but when C::B is loading my plugin on startup, C::B is not ready.

I need this check because:
1. the current C::B version doesn't post EVT_EDITOR_ACTIVATED for all recently-opened editors
2. wxExecute messes up GUI layout when called from within OnEditorActivated (I don't know why) on C::B startup.

[mistar]

OK
Sources of the very-very-very preliminary version of SemanticHighlight plugin are attached.
But be warned:
1. The plugin crashes sometimes as I've already mentioned. I suspect that this happens when SH does highlighting on a control and simultaneously the user types some text to that control. I don't know whether there is some way to lock the control during highlighting process, but maybe you know So please, do not use the plugin in its present form while doing important things in C::B!
2. If you change compile options of some target and/or project, SH doesn't notice those changes. It will when we add an event posted on such changes (EVT_PROJECT_TARGETS_MODIFIED is NOT posted on such changes though, in fact there is currently no event posted!).
3. There is no configuration panel yet.
4. On C::B startup, the GUI can be messed a little bit (wxExecute on EVT_EDITOR_ACTIVATED), but if you click on a boundary between panels everything returns to the correct layout.
5. Lines 301 and 302 of SemanticHighlight.cpp have to be adjusted according to your gcc configuration. Otherwise the plugin will work but some identifiers won't be highlighted correctly. These lines are needed in some gcc configurations as libclang doesn't include them in default search directories and then some fatal errors prevent libclang's parser to do entire job.
6. The plugin should work under Windows, but I didn't test it.
7. Please, check your compiler and linker options (libraries may have different names on your system).

Any questions and comments will be appreciated.

EDIT: I forgot: recall that the first parsing process may be quite long (about 4s on my machine).

[MortenMacFly]

Sources of the very-very-very preliminary version of SemanticHighlight plugin are attached.

To reach more tester, you should probably explain what dependencies you have, where they are available and how you "install" them.

[MortenMacFly]

2. If you change compile options of some target and/or project, SH doesn't notice those changes. It will when we add an event posted on such changes (EVT_PROJECT_TARGETS_MODIFIED is NOT posted on such changes though, in fact there is currently no event posted!).

I could think of a project options changed and compiler settings changed events, in case the project/compiler options are changed in the UI dialogs and the user hit "OK". For the compiler options this is, however, tricky. For example: You can attach build scripts to projects or even single files that evaluate at run-time. They may even using system calls to obtain specific settings. So you don't know the full compiler option for a single file until you build the file. Here, you should try to obtain these from the editor -> project file -> active target -> compiler options. You would, however, receive the current compiler options in the mentioned events which should work in 95 percent of the cases.
However, in the end that should be the same as you would obtain in editor activated event when the settings dialog was shown (same chain: editor -> project file -> active target -> compiler options).

[mistar]

Sources of the very-very-very preliminary version of SemanticHighlight plugin are attached.

To reach more tester, you should probably explain what dependencies you have, where they are available and how you "install" them.

To compile the plugin you need clang and llvm (see llvm.org) dynamic libraries installed.
The installation process should be quite easy, just install clang compiler and everything else should be installed as well
Then, adjust SH project's paths and libraries to match those just installed and everything should work.

2. If you change compile options of some target and/or project, SH doesn't notice those changes. It will when we add an event posted on such changes (EVT_PROJECT_TARGETS_MODIFIED is NOT posted on such changes though, in fact there is currently no event posted!).

I could think of a project options changed and compiler settings changed events, in case the project/compiler options are changed in the UI dialogs and the user hit "OK". For the compiler options this is, however, tricky. For example: You can attach build scripts to projects or even single files that evaluate at run-time. They may even using system calls to obtain specific settings. So you don't know the full compiler option for a single file until you build the file. Here, you should try to obtain these from the editor -> project file -> active target -> compiler options. You would, however, receive the current compiler options in the mentioned events which should work in 95 percent of the cases.

I know there can be very tricky ways for specifying compiler options. I use for example `wx-config --cflags` to obtain the compiler/linker options for wxWidgets and such tricky ways are supported in SH already.

However, in the end that should be the same as you would obtain in editor activated event when the settings dialog was shown (same chain: editor -> project file -> active target -> compiler options).

Calling my GetCompileOptions each time editor is activated is highly inefficient (several wxExecute calls to get options from commands like `wx-config --cflags`). Moreover, I avoid to process EVT_EDITOR_ACTIVATED each time the last active editor equals the current (windows switching for example), and there is (at present) no way for checking whether the options change at all, so I don't know whether to reparse sources or not I can, of course, reparse them each time EVT_EDITOR_ACTIVATED occurs but nevertheless it is worse solution IMHO.

I also debate whether to lock editor during first (several seconds long) reparsing, but there are some difficulties, for example I have no access to the percentage of the elapsed process, so I could only display some information box.
Please, share your opinions on that and similar things.

[MortenMacFly]

Moreover, I avoid to process EVT_EDITOR_ACTIVATED each time the last active editor equals the current (windows switching for example), and there is (at present) no way for checking whether the options change at all, so I don't know whether to reparse sources or not I can, of course, reparse them each time EVT_EDITOR_ACTIVATED occurs but nevertheless it is worse solution IMHO.

Yes, I think this is not really good. However, what about the other option(s) I suggested?

[mistar]

Moreover, I avoid to process EVT_EDITOR_ACTIVATED each time the last active editor equals the current (windows switching for example), and there is (at present) no way for checking whether the options change at all, so I don't know whether to reparse sources or not I can, of course, reparse them each time EVT_EDITOR_ACTIVATED occurs but nevertheless it is worse solution IMHO.

Yes, I think this is not really good. However, what about the other option(s) I suggested?

The events for project/target options changed and compiler options changed are IMHO the best solution here.
Since projects and targets are treated identically here (they are derived from CompileOptionsBase class) there might be one event kind.

And how about function returning true when C::B is starting and false otherwise? You have Manager::IsAppShuttingDown but no counterpart on startup.
If such a counterpart, say IsAppStartingUp, existed I could do all initializations in an event handler for EVT_APP_STARTUP when C::B is starting, in OnAttach when the plugin is loaded and in OnEditorActivated otherwise (I could then check in OnEditorActivated if IsAppStartingUp and do no initializations if so, as then all initializations would be done in OnAppStartup). This would solve, hopefully, the problem of messing up the GUI at C::B startup.

Also, did you consider adding a mutex for each wxScintilla control that would be publicly available? When I do highlighting I could aquire the mutex and then release it, and the same (internally in the control) when user types into the control. Now I have no tools to synchronize user input (which changes the layout of the control) and highlighting...

[MortenMacFly]

And how about function returning true when C::B is starting and false otherwise? You have Manager::IsAppShuttingDown but no counterpart on startup.

You can do it yourself easily: Have a global var "startupdone" that is false initially, connect to the "cbEVT_APP_STARTUP_DONE" event and set it to true if this fires. Combined with IsAppShuttingDown you get a method that "bools" if it is safe to operate or not.

Also, did you consider adding a mutex for each wxScintilla control that would be publicly available?

Well - that is not so easy. But what is wrong if you operate inside an event and call Skip() only if you have finished. This way you have the control exclusively during that time.

[mistar]

And how about function returning true when C::B is starting and false otherwise? You have Manager::IsAppShuttingDown but no counterpart on startup.

You can do it yourself easily: Have a global var "startupdone" that is false initially, connect to the "cbEVT_APP_STARTUP_DONE" event and set it to true if this fires. Combined with IsAppShuttingDown you get a method that "bools" if it is safe to operate or not.

I cannot do it in the plugin, it has to be provided from outside, that is from the main application. The reason is that attaching a plugin may occur after EVT_APP_STARTUP_DONE is fired (loading plugin via Plugins->Manage plugins).

Also, did you consider adding a mutex for each wxScintilla control that would be publicly available?

Well - that is not so easy. But what is wrong if you operate inside an event and call Skip() only if you have finished. This way you have the control exclusively during that time.

The problem is that I do not want to block application until reparse-highlight process is completed, which may happen long time after OnTextChanged handler returns (I have threads, one for each editor for parsing/highlighting). Second, reparsing is much longer than highlighting and I need to lock the editor's control only for highlighting.
IMHO, Scintilla should take care of synchronizing access to the control, so I'll try to use one function (wxScintilla::SetStyleBytes) to do highlighting and maybe this helps.

[mistar]

There are tools that should help in locking a mutex for an editor while updating the editor's control, but they do not work properly...
The solution would be to handle wxEVT_SCI_MODIFIED in the following way:
* if GetModificationType() is SC_MOD_BEFOREINSERT/SC_MOD_BEFOREDELETE (-> http://www.scintilla.org/ScintillaDoc.html#Notifications) then Lock the editor's mutex;
* if GetModificationType() is SC_MOD_INSERTTEXT/SC_MOD_DELETETEXT (which, according to the documentation, should occur after the control's modification) then Unlock the editor's mutex and do reparsing-highlighting;

However, I observed that updating control occurs after SC_MOD_INSERTTEXT/SC_MOD_DELETETEXT is handled (except for the first modification after startup...). Maybe I'm wrong and this is only delay in GUI redrawing...

I'm attaching sources. Note the lines 388-390 and 401-402 with wxSleep time-margins. To reproduce the behavior, just open C::B from command line so that you see the stderr output, place C::B with the terminal visible behind. The sequence in the terminal should be:

Code

*** before
(500 millisecond pause)
**** update
(control update)
(2000 millisecond pause)
*** after

but is

Code

*** before
(500 millisecond pause)
**** update
(2000 millisecond pause)
*** after and control update simultaneously

As for 'bool IsAppStartingUp()' function, shall I provide a patch?
If so, please guide me where to define the global variable 'bool isAppStarted' or so.
I understand that it should be set to true at the end of C::B's OnInit().
But it would be easier and faster if someone of C::B team could do it for me

[MortenMacFly]

As for 'bool IsAppStartingUp()' function, shall I provide a patch?

Yes, just do it similar to the IsShuttingDown method. I'm off the next few days - so you have enough time for a patch... Otherwise I can do if I'm back ;-)

[mistar]

As for 'bool IsAppStartingUp()' function, shall I provide a patch?

Yes, just do it similar to the IsShuttingDown method. I'm off the next few days - so you have enough time for a patch... Otherwise I can do if I'm back ;-)

Ok, I'll try to do it myself.

I also checked the exact order of scintilla's events and UI update and it is

Code

wxEVT_SCI_MODIFIED(BEFORE_INSERT/DELETE)
UI (control) update
wxEVT_SCI_MODIFIED(TEXTINSERT/DELETE)

the first time editor is activated and

Code

wxEVT_SCI_MODIFIED(BEFORE_INSERT/DELETE)
wxEVT_SCI_MODIFIED(TEXTINSERT/DELETE)
UI (control) update

each subsequent time. Really weird...
Is this scintilla's bug, or C::B? Or maybe scintilla buffers graphical updates on several inserts/deletes?

Nevertheless the plugin hasn't crashed since yesterday after adding some synchronization

Now I'm going to do the most stupid part, that is, to write configuration panel...