Author Topic: Semantic highlight  (Read 94243 times)

Offline mistar

  • Multiple posting newcomer
  • *
  • Posts: 42
Re: Semantic highlight
« Reply #15 on: May 15, 2012, 11:03:03 am »
Hello mistar, may i ask what functions the plugin will be offered?

Basically, it should highlight the c++ syntax in a semantic way.
For example, in highlighting based on lexers, all identifiers have the same color.
In semantic highlight on the other hand, variables, types, functions, etc. are highlighted in different ways.

If you work for some time with semantic highlight and then switch back to the usual, lexer-based highlight,
you notice the difference as if you switch from lexer-based highlight to no highlight at all ;).

Some tools provide semantic highlight, eg. Eclipse CDT or KDevelop, but they lack a support for the new C++11 standard.
Also, both have internal parsers that are not developed to support the new standard.
Since libclang (see http://clang.llvm.org; nb. version 3.1 is about to be released) has the best support for the new standard
and it provides parser API, it is relatively easy to write a plugin for C::B.

So far I have written a very simple tool that highlights a source in a semantic way (so it can be done using manual styling in Scintilla).
Now I'm working on retrieving compiler options (include directories, defines, etc.) so that all declarations are accessible to the libclang parser
(otherwise it highlights fragments of code as invalid).
Since some options are given implicitly (eg. via `wx-config --cxxflags`) I'm using wxExecute to retrieve the options, but it messes up the C::B user interface
on C::B sturtup (it seems that wxAuiManager and wxExecute interfere in some way)... any ideas how to do this in another way?
(The output of C::B on stdout contains "caching results from `wx-config --cxxflags`", so maybe I can use this cache?)
I've tried also to do this in a separate thread but the thread freezes (perhaps wxExecute in not thread-safe).

@MortenMacFly: Thanks, I'll have a look.

Offline perento

  • Multiple posting newcomer
  • *
  • Posts: 47
Re: Semantic highlight
« Reply #16 on: May 15, 2012, 04:44:36 pm »
Nice nice, i was looking for something to highlight user functions, classes, structures etc. and i was leaded to you thread.
Would like to see that plugin how it works, keep the work please :)

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: Semantic highlight
« Reply #17 on: May 15, 2012, 08:49:02 pm »
(it seems that wxAuiManager and wxExecute interfere in some way)...
They have nothing to do with each other.

any ideas how to do this in another way?
Have a look at other plugins that run external tools (e.g. CppCheck, CScope or alike), you can see how its done there.

(The output of C::B on stdout contains "caching results from `wx-config --cxxflags`", so maybe I can use this cache?)
This information is not available via a public interface.

I've tried also to do this in a separate thread but the thread freezes (perhaps wxExecute in not thread-safe).
We use threads a lot in C::B if yours freezes then you did something wrong. You can even use the thread abstractions available via the SDK for such purposes. Again: Have a look how other plugins do it (i.e. ThreadSearch).
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline mistar

  • Multiple posting newcomer
  • *
  • Posts: 42
Re: Semantic highlight
« Reply #18 on: May 16, 2012, 08:11:45 pm »
Hi guys!

Before my work on SemanticHighlight plugin goes on, let me introduce simple SHTool plugin I've just written
to demonstrate what can be achieved. The sources are attached.

This is a very simple tool plugin that colorize the syntax in a semantic way using hardcoded styles that I like
(see the sources).
To revert the styles to original ones just reopen the file.
I've tested it with C::B 10.05 and with latest svn version (with change of sdk version in manifest.xml to 1.13.2).
The plugin works best with project files (with all compiler options and search directories set up correctly).
The plugin links against libclang.so so you have to install clang first.
Also note that parsing a file from scratch takes some time (when editing a file there are some optimizations
so in the final version it should take no more than a second).

Have fun!
« Last Edit: May 16, 2012, 08:13:23 pm by mistar »

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Semantic highlight
« Reply #19 on: May 16, 2012, 10:21:37 pm »
Also note that parsing a file from scratch takes some time (when editing a file there are some optimizations
so in the final version it should take no more than a second).
A second for every key press? :)
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline mistar

  • Multiple posting newcomer
  • *
  • Posts: 42
Re: Semantic highlight
« Reply #20 on: May 16, 2012, 11:30:34 pm »
A second for every key press? :)
A second after last key press.
In eclipse CDT it's not a problem; if it is done in separate thread then only visual results are delayed a bit.
Besides, there is no way to do this immediately. ;)
Anyway, in my opinion benefits surpass all the inconveniences.
« Last Edit: May 16, 2012, 11:33:22 pm by mistar »

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Semantic highlight
« Reply #21 on: May 17, 2012, 03:29:14 am »
Hi guys!

Before my work on SemanticHighlight plugin goes on, let me introduce simple SHTool plugin I've just written
to demonstrate what can be achieved. The sources are attached.

This is a very simple tool plugin that colorize the syntax in a semantic way using hardcoded styles that I like
(see the sources).
To revert the styles to original ones just reopen the file.
I've tested it with C::B 10.05 and with latest svn version (with change of sdk version in manifest.xml to 1.13.2).
The plugin works best with project files (with all compiler options and search directories set up correctly).
The plugin links against libclang.so so you have to install clang first.
Also note that parsing a file from scratch takes some time (when editing a file there are some optimizations
so in the final version it should take no more than a second).

Have fun!
Hi, nice work.
I just build it under Windows. Here is what I do:
1, Put your source and package under:
Code
E:\code\cb\cb_trunk\src\plugins\contrib\SHTool

2, I borrow the clang from Codelite source repo:
Code
https://codelite.svn.sourceforge.net/svnroot/codelite/trunk/sdk/clang
And put the folder "clang" in the folder "SHTool"

3, I have create a cbp file for Windows called:SHTool_Win.cbp (see attachment)

4, open the SHTool_Win.cbp, and build it. (the output will be put in E:\code\cb\cb_trunk\src\devel)

5, run the Codeblocks under E:\code\cb\cb_trunk\src\devel, Make sure the clang.dll is in your PATH.

The problem is: I can't see this plugin works when I open a new project, I can see the SHTool plugin is loaded. :(
The editor is not highlighted by SHTool.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Semantic highlight
« Reply #22 on: May 17, 2012, 03:40:15 am »
The problem is: I can't see this plugin works when I open a new project, I can see the SHTool plugin is loaded. :(
The editor is not highlighted by SHTool.
Ok, it works now, I need to click the Menu->Plugins->SHTool, it works really NICE.

See the screen shot with semantic highlight below:

Compared with normal editor
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Semantic highlight
« Reply #23 on: May 17, 2012, 08:52:52 am »
Ollydbg: This is not a demonstration of semantic highlight... you have to show a function's body...
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline mistar

  • Multiple posting newcomer
  • *
  • Posts: 42
Re: Semantic highlight
« Reply #24 on: May 17, 2012, 09:28:32 am »
The problem is: I can't see this plugin works when I open a new project, I can see the SHTool plugin is loaded. :(
The editor is not highlighted by SHTool.
Ok, it works now, I need to click the Menu->Plugins->SHTool, it works really NICE.
Yes, now this is only a simple tool (when you want to highlight sources just Menu->Plugins->SHTool; I forgot to write it explicitly), not real-time highlighter.
Now it is time to design a framework for real-time SemanticHighlight plugin.
Just a couple of event handlers, provided the events are fired in right order and with all required data... ;)

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Semantic highlight
« Reply #25 on: May 17, 2012, 10:22:02 am »
mistar:
Is it possible to make the required changes in the SDK, so multiple plugins can be semantic highlight providers?
I guess if we make a CC plugin based on clang we have to implement the semantic highlight in it in order to prevent calling clang to parse the code twice.

Is the parsing required for CC and semantic parsing the same?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline mistar

  • Multiple posting newcomer
  • *
  • Posts: 42
Re: Semantic highlight
« Reply #26 on: May 17, 2012, 10:31:48 am »
mistar:
Is it possible to make the required changes in the SDK, so multiple plugins can be semantic highlight providers?
I guess if we make a CC plugin based on clang we have to implement the semantic highlight in it in order to prevent calling clang to parse the code twice.

Is the parsing required for CC and semantic parsing the same?
AFAIK, clang_codeCompleteAt calls clang_reparseTranslationUnit, see the description of clang_defaultEditingTranslationUnitOptions here: http://clang.llvm.org/doxygen/group__CINDEX__TRANSLATION__UNIT.html

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Semantic highlight
« Reply #27 on: May 17, 2012, 10:40:43 am »
I've never looked at the clang's API and I restrain myself from doing so, because I'd be caught in a nasty CC refactoring.  :P
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline mistar

  • Multiple posting newcomer
  • *
  • Posts: 42
Re: Semantic highlight
« Reply #28 on: May 19, 2012, 11:28:57 am »
I think there is a better way: Instead of introducing another event, you could move the portion:
Code
    if (can_updateui)
    {
        if (ed)
        {
            SetActiveEditor(ed);
            ed->GetControl()->SetFocus();
        }
    }
...to after:
Code
    // check for ProjectFile
    if (ed && !ed->GetProjectFile())
    {
        // First checks if we're already being passed a ProjectFile as a parameter
        if (data)
            Manager::Get()->GetLogManager()->DebugLog(_T("Project data set for ") + data->file.GetFullPath());
        else
            Manager::Get()->GetProjectManager()->FindProjectForFile(ed->GetFilename(), &data, false, false);
        if (data)
            ed->SetProjectFile(data,true);
    }
This way, you can use the already existing cbEVT_EDITOR_ACTIVATED event and don't screw anything.

I'll check this in my local copy and look for side-effects (but I cannot think of any for the moment...).

Well, this is not what I need... My understanding of cbEVT_EDITOR_ACTIVATED is that it is posted whenever an editor gets focus...
What I really need is an event that is posted when a file is opened in C::B to set up clang data (index, translation unit, ...) for that file
AND containing all necessary data like the project the file belongs to (to read the project's compiler options needed to correctly set up the index).

BTW, it is good practice to name things adequately so that others can understand our code ;) Therefore, if cbEVT_EDITOR_OPEN is reserved for
"is about to be opened" (and there is no extra data set up at the moment), then I believe that there might (or even should) be another event, e.g. cbEVT_EDITOR_LOADED
(this is probably better name) or whatever, signaling that the opening action is completed with all the data set up correctly.
Observe also, that my implementation shouldn't screw anything: it is just a piece of code added to what exists and this code shouldn't interfere with
the existing code.

Last but not least, cbEVT_PROJECT_OPEN event is not semantically "is about to be opened" ;)

EDIT: I would need also an event notifying that options in a build target or in a project have changed to update the highlighter's data...
The existing cbEVT_PROJECT_TARGETS_MODIFIED is not posted when I change compiler (or other) options in a target or project (in fact no event is posted at all).
The solution of reading compiler options from the active build target every time highlighter is invoked is not elegant and poses problems with wxExecute that I already mentioned
(that is, on C::B startup, wxExecute in OnEditorOpen event handler messes up the UI: image attached; I observed that loading some plugin or changing a little bit UI fixes the layout).

[attachment deleted by admin]
« Last Edit: May 19, 2012, 01:17:02 pm by mistar »

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: Semantic highlight
« Reply #29 on: May 19, 2012, 01:52:20 pm »
I'm afraid I don't really understand. When you load a project, C::B does not load all files, so there is no such event you are asking for. If you are interested in parsing all files when the project is opened, you need the EVT_PROJECT_OPEN or better EVT_PROJECT_ACTIVATE. Then, in addition you can use EVT_PROJECT_TARGETS_MODIFIED, EVT_PROJECT_FILE_CHANGED, EVT_PROJECT_FILE_ADDED and EVT_PROJECT_END_REMOVE_FILES to check for changes in the setup. There is EVT_SETTINGS_CHANGED for checking, if the project's settings have been changed.

However, it makes no sense to prepare semantic highlighting for files that are actually not really opened, a.k.a. shown as an editor. Imagine you have workspaces with several 1000 files - how long shall the user wait? So semantic highlight IMHO gets interesting, once you see a file in an editor, that's why I pointed you to the EVT_EDITOR_ACTIVATED event.

Have a look at the Code::Completion plugin. It does all that: Checking for project settings, checking for file changes, running external tools (like the compiler to query the compiler's internal settings), workign once a file is opened and so on. CC would not work if all that was not possible.

If you still believe you need more, provide patches. The one you provided does not help you much, it is the same as the EVT_EDITOR_ACTIVATED event, once you did the changes as I suggested. Maybe I'll understand better your needs if you show what you have in mind.

For the wxExecute issue: Run it as an external process, use the C::B SDK to do so, maybe even in a thread. then you won't have such issues. It works really nice in several plugins we already ave, so I see no reason why it should not work for you, except you might have done something wrong. Again: in that case provide a test case / sample to reproduce.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ