Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => Topic started by: Malc on July 11, 2012, 11:37:11 pm

Title: Extending the CPP Lexer
Post by: Malc on July 11, 2012, 11:37:11 pm
I've looked at the code, and it looks as simple as just adding another line to cppWordLists[] in LexCPP.cxx, but I wanted to check first.  I need support for a 3rd set of Keywords.  Using the existing CPP Lexer, I have found I can highlight comments, preprocessor stuff (index 9), primary keywords (index 0) like bool int float etc, and secondary keywords.  I can't get a third set working though within the bounds of the existing lexer.

The purpose of this is for another c-type language that has many global functions, as well as local functions, and I want them highlighted separately.  The language is LPC, in case someone already has support for it  ;D which would be even better.  I want to switch from VIM to CB ;D

Thank you,
Malc
Title: Re: Extending the CPP Lexer
Post by: MortenMacFly on July 12, 2012, 07:43:54 am
Did you try menu "Settings" -> "Editor" -> "Syntax Highlighting" -> Button "Keywords" first? Here you can setup additional user keywords for each (!) lexer. I think this is way better than hacking scintilla (the lexer's code) itself.
Title: Re: Extending the CPP Lexer
Post by: Alpha on July 12, 2012, 05:42:27 pm
I think the easiest (and probably best) way to support this language would be to copy lexer_cpp.xml to lexer_lcp.xml and modify the file masks and keywords.  However, this would still only give you two color sets for highlighted keywords, and if I am understanding you correctly, you want three, right?
Title: Re: Extending the CPP Lexer
Post by: Malc on July 13, 2012, 06:12:12 am
Quote from: MortenMacFly on July 12, 2012, 07:43:54 am
Did you try menu "Settings" -> "Editor" -> "Syntax Highlighting" -> Button "Keywords" first? Here you can setup additional user keywords for each (!) lexer. I think this is way better than hacking scintilla (the lexer's code) itself.

Yes, this would be easier, but still does not allow for a 3rd set.  For example, you can enter 1-9.  But if I use '4', as an example, those words are not highlighted.

Quote from: Alpha on July 12, 2012, 05:42:27 pm
I think the easiest (and probably best) way to support this language would be to copy lexer_cpp.xml to lexer_lcp.xml and modify the file masks and keywords.  However, this would still only give you two color sets for highlighted keywords, and if I am understanding you correctly, you want three, right?
Correct.  I started out this way, experimenting with indexes and such as well.  But I cannot get 3 sets of keywords :(
Title: Re: Extending the CPP Lexer
Post by: Alpha on July 13, 2012, 11:05:51 pm
In that case, modifying LexCPP.cxx might be your only option.  If you go through with this, you also should talk to the Scintilla (http://www.scintilla.org/) people to try to get it in their repository.  (Code::Blocks uses a mostly unmodified copy of Scintilla.)
Title: Re: Extending the CPP Lexer
Post by: Malc on July 14, 2012, 12:04:49 am
So if I do this, what would need to be modified?  It looks to me like just the wordlist array.  Anything else?
Title: Re: Extending the CPP Lexer
Post by: Folco on July 14, 2012, 09:16:36 am
If you want to add a lexer, you have to modify :
src/sdk/wxscintilla/src/scintilla/src/Catalogue.cxx
src/sdk/wxscintilla/Makefile.am
src/sdk/wxscintilla/src/scintilla/include/SciLexer.h
src/sdk/wxscintilla/src/scintilla/include/Scintilla.iface
src/sdk/wxscintilla/include/wx/wxscintilla.h

In your lexer_language.cxx, you must define the entry points of the keyword list that you want to use (indexes are here !). Example :
    WordList &cpuInstruction = *keywordlists[0];
    WordList &registers      = *keywordlists[1];
    WordList &directive      = *keywordlists[2];
    WordList &extInstruction = *keywordlists[3];
    WordList &alert          = *keywordlists[4];
    WordList &doxygenKeyword = *keywordlists[5];

Then you must export that, in the same source file :
static const char * const a68kWordListDesc[] =
{
    "CPU instructions",
    "Registers",
    "Directives",
    "Extended instructions",
    "Alerts",
    "Doxygen keywords",
    0
};

That is the right way to define new indexes and keyword lists.
Title: Re: Extending the CPP Lexer
Post by: MortenMacFly on July 14, 2012, 10:14:31 am
... I would seriously suggest you first contact and discuss any changes with the maintainer of scintilla (http://www.scintilla.org/ScintillaToDo.html). He is known to be very restrictive. And what won't make it into scintilla itself most likely also won't make it into C::B.
Title: Re: Extending the CPP Lexer
Post by: Malc on July 15, 2012, 05:23:42 am
So Folco, are those first file modifications needed if you are modifying an existing lexer?  Should just be the extension of the wordlists themselves, correct?

Morten, unless the Scintilla maintainer is willing to add the changes in himself, I'm not really concerned about it.  I can just compile it myself and call it good.

:)
Malc
Title: Re: Extending the CPP Lexer
Post by: Alpha on July 15, 2012, 06:56:25 am
There are other modifications.  Follow the variable keywords2 around LexCPP.cxx and do a threaded searches for SCE_C_WORD2 and wxSCI_C_WORD2.  That should provide an example, and the locations for modifications.
Title: Re: Extending the CPP Lexer
Post by: Folco on July 15, 2012, 09:03:21 am
Malc -> perhaps you have not to modify all these files. Look into them. I don't remember exactly because I rewrote a lexer from scratch for me, and you just want to modify an existing one.
Title: Re: Extending the CPP Lexer
Post by: Malc on July 16, 2012, 12:36:18 am
Quote from: Alpha on July 15, 2012, 06:56:25 am
There are other modifications.  Follow the variable keywords2 around LexCPP.cxx and do a threaded searches for SCE_C_WORD2 and wxSCI_C_WORD2.  That should provide an example, and the locations for modifications.

Quote from: Folco on July 15, 2012, 09:03:21 am
Malc -> perhaps you have not to modify all these files. Look into them. I don't remember exactly because I rewrote a lexer from scratch for me, and you just want to modify an existing one.

Thank you both!
Malc
Title: Re: Extending the CPP Lexer
Post by: Alpha on July 17, 2012, 06:18:30 pm
Wait!  No programming of the lexer necessary!  Add:
Code
                <Style name="Global classes and typedefs"
                       index="19"
                       fg="190,0,190"
                       bold="1"/>
to lexer_cpp.xml, then list your words in
Code
                        <Set index="3"
                             value="my list of keywords"/>
or in the set labeled "4" inside the highlighting settings.
Title: Re: Extending the CPP Lexer
Post by: carra on July 18, 2012, 10:02:47 pm
Thanks Alpha! Works like a charm. Very interesting for highlighting domain vocabulary in custom projects.
Title: Re: Extending the CPP Lexer
Post by: Malc on July 19, 2012, 07:49:15 am
Quote from: Alpha on July 17, 2012, 06:18:30 pm
Wait!  No programming of the lexer necessary!  Add:

AWESOME!  Thank you Alpha, that works awesomely!  I am at a loss why we need to deal with 3 different numbers (19,3,4) for indexing, but nevertheless, very nice :)


I don't suppose you know to change something like:

#define SOMEVAR(x) somefunc(x)
to just be
#define SOMEVAR(x) somefunc(x)

It seems to want to highlight the whole line.
Title: Re: Extending the CPP Lexer
Post by: Alpha on July 19, 2012, 10:43:55 pm
Here is the quick hack.  (To do it properly, this should actually be connected to a checkbox in the settings.)
Code
Index: src/sdk/wxscintilla/src/scintilla/lexers/LexCPP.cxx
===================================================================
--- src/sdk/wxscintilla/src/scintilla/lexers/LexCPP.cxx	(revision 8148)
+++ src/sdk/wxscintilla/src/scintilla/lexers/LexCPP.cxx	(working copy)
@@ -221,7 +221,7 @@
 	bool highlightWxSmith;
 /* C::B end */
 	OptionsCPP() {
-		stylingWithinPreprocessor = false;
+		stylingWithinPreprocessor = true;
 		identifiersAllowDollars = true;
 		trackPreprocessor = true;
 		updatePreprocessor = true;
Title: Re: Extending the CPP Lexer
Post by: MortenMacFly on July 20, 2012, 02:22:14 pm
Quote from: Alpha on July 19, 2012, 10:43:55 pm
(To do it properly, this should actually be connected to a checkbox in the settings.)
Which are roughly 3 lines of code and a checkbox in a XRC file you can modify with C::B. Patches are welcome. (Not from you, Alpha - you've done enough already...)

Edit: See:
http://svn.berlios.de/wsvn/codeblocks/branches/xml_compiler/?op=revision&rev=8136&peg=8136
(Only the files editor_configuration.xrc, editorconfigurationdlg.cpp and cbeditor.cpp)