Author Topic: CC based semantic highlight  (Read 18004 times)

Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1513
CC based semantic highlight
« on: February 16, 2013, 01:12:30 am »
I have begun exploration of full semantic highlighting, provided by the knowledge from CC's parser.  The attached patch is able to show colors, however it is very temperamental.  Also, if you type anything into the editor, it will break.
The colors were chosen at random; suggestions for these would be appreciated.

This code currently uses some operations which are potentially unsafe.  You have been warned.


Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5339
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: CC based semantic highlight
« Reply #1 on: February 16, 2013, 02:18:16 am »
Great work. Look at the screen shot, it is so beautiful.
I'm going to test it right now.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5339
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: CC based semantic highlight
« Reply #2 on: February 16, 2013, 04:52:45 pm »
Testing result: sometimes, C::B hangs. :)
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline mistar

  • Multiple posting newcomer
  • *
  • Posts: 42
Re: CC based semantic highlight
« Reply #3 on: February 20, 2013, 10:12:48 am »

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5339
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: CC based semantic highlight
« Reply #4 on: February 20, 2013, 03:07:48 pm »
Have a look also on my plugin ;) shouldn't hang ;)
http://forums.codeblocks.org/index.php/topic,16249.msg120205.html#msg120205
Both you and Alpha did good job, thanks, will take time to test those.

@Alpha, about the hang, I think the locker(there are many lockers in the CC code), but from my point of view, we should remove mostly of them.

TokenTree are the shared resource, sometimes, the worker thread (Parserthread are adding tokens to it), in the other time, the main GUI thread(like the one you implement the semantic highlight should query the TokenTree). I think they should not happen concurrently. I mean if the parser is working, the TokenTree should not accessed by main thread. Either main GUI thread or worker thread can access the TokenTree.

If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline p2rkw

  • Almost regular
  • **
  • Posts: 142
Re: CC based semantic highlight
« Reply #5 on: March 30, 2013, 01:31:34 am »
I see it's in trunk now: http://sourceforge.net/p/codeblocks/code/8920/
but as I can see it's impossible to disable this feature.
My proposition is to detect whenever third keyword set have exactly the same colour as default keyword set: if those colours are equal then all token processing inside CodeCompletion::UpdateEditorSyntax is unnecessary.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13210
    • Travis build status
Re: CC based semantic highlight
« Reply #6 on: March 30, 2013, 11:48:51 am »
Alpha:
Do you plan to fix this issue:
Code
class Test
{
    void func()
    {
        int a = var;
    }
    int var;
};

void func()
{
    int var;
}
Hint: the issue is that the var is highlighted everywhere, but it shouldn't inside the func.

If not please revert your patch!
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1513
Re: CC based semantic highlight
« Reply #7 on: March 30, 2013, 07:13:50 pm »
The one that is in trunk is actually this.  But I will add an enable/disable checkbox now.

Do you plan to fix this issue: [...]
Impossible with the current method of implementation (see other thread).  Just disable it, if it bothers you, when I get the checkbox uploaded.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13210
    • Travis build status
Re: CC based semantic highlight
« Reply #8 on: March 30, 2013, 07:49:32 pm »
Impossible with the current method of implementation (see other thread).  Just disable it, if it bothers you, when I get the checkbox uploaded.
I'll just revert it, don't worry. There is no point in having a fake feature in trunk. If you want it in do it properly, please.

@other devs: please state your opinion here, I won't revert if for a couple of days.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5339
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: CC based semantic highlight
« Reply #9 on: March 31, 2013, 08:16:53 am »
C++ is too complex and its grammar is context dependant, the way Alpha use was set the identifier to a group, then let the scintilla to colorize the group in the same color. I think we can follow what the way clang sematic highlight plugin use, I mean write our own lexer for scintilla, colorize each piece of text by the token type. The token type info can be queryed from the tokentree. But this may still cause error, because our parser does not do sematic check.

EDIT: It looks that it is impossible to query every text info from the tokentree, our tokentree only hold the variable definition info, but for other variable usage(reference), it need the AI() function, this will make the custom lexer much slower.
« Last Edit: March 31, 2013, 04:09:36 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9613
Re: CC based semantic highlight
« Reply #10 on: March 31, 2013, 05:23:50 pm »
@other devs: please state your opinion here, I won't revert if for a couple of days.
Well I like and in fact use it although I realise its not working 100%. Maybe this should be turned off by default and an explanation in the settings points out that this is not meant to work perfectly.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13210
    • Travis build status
Re: CC based semantic highlight
« Reply #11 on: March 31, 2013, 05:49:01 pm »
Well I like and in fact use it although I realise its not working 100%. Maybe this should be turned off by default and an explanation in the settings points out that this is not meant to work perfectly.
For sure it should be turned off by default if it stays.
But the problem is that this is a half feature - broken and hard to extend.

The thing is that the semantic part of the name of the feature implies that it uses the CC and so it should work 100% of the time (ignoring CC parser bugs).
But this is not the case. And we have a prototype (I've not looked at it, so I cannot tell how good it is) that does better job.

Also this feature is almost the same as the broken highlight for stl stuff, but the brokenness is a bit more hidden.

I know that implementing this properly is hard, because of the need to modify scintilla, but I'm sure it is doable.
Also the semantic highlight should not be part of the CC plugin! It should be part of the SDK or it should be separated in a plugin.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1513
Re: CC based semantic highlight
« Reply #12 on: April 04, 2013, 12:58:07 am »
It is off by default now.  If you would like to revert it completely, go ahead, however from previous discussions of this patch, it sounded as though there are at least several people who do find it useful in its current (flawed) state.

Also the semantic highlight should not be part of the CC plugin! It should be part of the SDK or it should be separated in a plugin.
Agreed, but do we yet have a way for a seperate plugin to access the data in CC's parser?

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13210
    • Travis build status
Re: CC based semantic highlight
« Reply #13 on: April 04, 2013, 08:19:06 am »
It is off by default now.  If you would like to revert it completely, go ahead, however from previous discussions of this patch, it sounded as though there are at least several
people who do find it useful in its current (flawed) state.
OK, but what do you do when they ask you to highlight the member functions differently and then the local variables?

Agreed, but do we yet have a way for a seperate plugin to access the data in CC's parser?
Nope, but we should start from somewhere otherwise we will never have it.

BTW: The current implementation is quite laggy and thus quite annoying. Something happens but I don't know when.

p.s. Anyone else from the devs which wants to express opinion. Currently we are 1:1.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9613
Re: CC based semantic highlight
« Reply #14 on: April 04, 2013, 07:40:23 pm »
p.s. Anyone else from the devs which wants to express opinion. Currently we are 1:1.
Express opinion on what exactly?

For semantic highlight and access to CC internals my preferences are to use clang, maybe through Eran's engine. I think we can never be better than that - its amazing how fast the code base and functional support grows. So having a nice abstraction layer for clang will serve way more than providing interfaces to access (our) CC internals IMHO...
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ