Author Topic: Extending the CPP Lexer (Read 19176 times)

Malc · « **on:** July 11, 2012, 11:37:11 pm »

I've looked at the code, and it looks as simple as just adding another line to cppWordLists[] in LexCPP.cxx, but I wanted to check first. I need support for a 3rd set of Keywords. Using the existing CPP Lexer, I have found I can highlight comments, preprocessor stuff (index 9), primary keywords (index 0) like bool int float etc, and secondary keywords. I can't get a third set working though within the bounds of the existing lexer.

The purpose of this is for another c-type language that has many global functions, as well as local functions, and I want them highlighted separately. The language is LPC, in case someone already has support for it

which would be even better. I want to switch from VIM to CB

Thank you,
Malc

MortenMacFly · « **Reply #1 on:** July 12, 2012, 07:43:54 am »

Did you try menu "Settings" -> "Editor" -> "Syntax Highlighting" -> Button "Keywords" first? Here you can setup additional user keywords for each (!) lexer. I think this is way better than hacking scintilla (the lexer's code) itself.

Alpha · « **Reply #2 on:** July 12, 2012, 05:42:27 pm »

I think the easiest (and probably best) way to support this language would be to copy lexer_cpp.xml to lexer_lcp.xml and modify the file masks and keywords. However, this would still only give you two color sets for highlighted keywords, and if I am understanding you correctly, you want three, right?

Malc · « **Reply #3 on:** July 13, 2012, 06:12:12 am »

Quote from: MortenMacFly on July 12, 2012, 07:43:54 am

Did you try menu "Settings" -> "Editor" -> "Syntax Highlighting" -> Button "Keywords" first? Here you can setup additional user keywords for each (!) lexer. I think this is way better than hacking scintilla (the lexer's code) itself.

Yes, this would be easier, but still does not allow for a 3rd set. For example, you can enter 1-9. But if I use '4', as an example, those words are not highlighted.

Quote from: Alpha on July 12, 2012, 05:42:27 pm

I think the easiest (and probably best) way to support this language would be to copy lexer_cpp.xml to lexer_lcp.xml and modify the file masks and keywords. However, this would still only give you two color sets for highlighted keywords, and if I am understanding you correctly, you want three, right?

Correct. I started out this way, experimenting with indexes and such as well. But I cannot get 3 sets of keywords

Alpha · « **Reply #4 on:** July 13, 2012, 11:05:51 pm »

In that case, modifying LexCPP.cxx might be your only option. If you go through with this, you also should talk to the Scintilla people to try to get it in their repository. (Code::Blocks uses a mostly unmodified copy of Scintilla.)

Malc · « **Reply #5 on:** July 14, 2012, 12:04:49 am »

So if I do this, what would need to be modified? It looks to me like just the wordlist array. Anything else?

Folco · « **Reply #6 on:** July 14, 2012, 09:16:36 am »

If you want to add a lexer, you have to modify :
src/sdk/wxscintilla/src/scintilla/src/Catalogue.cxx src/sdk/wxscintilla/Makefile.am src/sdk/wxscintilla/src/scintilla/include/SciLexer.h src/sdk/wxscintilla/src/scintilla/include/Scintilla.iface src/sdk/wxscintilla/include/wx/wxscintilla.h

In your lexer_language.cxx, you must define the entry points of the keyword list that you want to use (indexes are here !). Example :
WordList &cpuInstruction = *keywordlists[0]; WordList &registers = *keywordlists[1]; WordList &directive = *keywordlists[2]; WordList &extInstruction = *keywordlists[3]; WordList &alert = *keywordlists[4]; WordList &doxygenKeyword = *keywordlists[5];

Then you must export that, in the same source file :
static const char * const a68kWordListDesc[] = { "CPU instructions", "Registers", "Directives", "Extended instructions", "Alerts", "Doxygen keywords", 0 };

That is the right way to define new indexes and keyword lists.

MortenMacFly · « **Reply #7 on:** July 14, 2012, 10:14:31 am »

... I would seriously suggest you first contact and discuss any changes with the maintainer of scintilla (http://www.scintilla.org/ScintillaToDo.html). He is known to be very restrictive. And what won't make it into scintilla itself most likely also won't make it into C::B.

Malc · « **Reply #8 on:** July 15, 2012, 05:23:42 am »

So Folco, are those first file modifications needed if you are modifying an existing lexer? Should just be the extension of the wordlists themselves, correct?

Morten, unless the Scintilla maintainer is willing to add the changes in himself, I'm not really concerned about it. I can just compile it myself and call it good.

Malc

Alpha · « **Reply #9 on:** July 15, 2012, 06:56:25 am »

There are other modifications. Follow the variable keywords2 around LexCPP.cxx and do a threaded searches for SCE_C_WORD2 and wxSCI_C_WORD2. That should provide an example, and the locations for modifications.

Folco · « **Reply #10 on:** July 15, 2012, 09:03:21 am »

Malc -> perhaps you have not to modify all these files. Look into them. I don't remember exactly because I rewrote a lexer from scratch for me, and you just want to modify an existing one.

Malc · « **Reply #11 on:** July 16, 2012, 12:36:18 am »

Quote from: Alpha on July 15, 2012, 06:56:25 am

There are other modifications. Follow the variable keywords2 around LexCPP.cxx and do a threaded searches for SCE_C_WORD2 and wxSCI_C_WORD2. That should provide an example, and the locations for modifications.

Quote from: Folco on July 15, 2012, 09:03:21 am

Malc -> perhaps you have not to modify all these files. Look into them. I don't remember exactly because I rewrote a lexer from scratch for me, and you just want to modify an existing one.

Thank you both!
Malc

Alpha · « **Reply #12 on:** July 17, 2012, 06:18:30 pm »

Wait! No programming of the lexer necessary! Add:

Code

                <Style name="Global classes and typedefs"
                       index="19"
                       fg="190,0,190"
                       bold="1"/>

to lexer_cpp.xml, then list your words in

Code

                        <Set index="3"
                             value="my list of keywords"/>

or in the set labeled "4" inside the highlighting settings.

carra · « **Reply #13 on:** July 18, 2012, 10:02:47 pm »

Thanks Alpha! Works like a charm. Very interesting for highlighting domain vocabulary in custom projects.

Malc · « **Reply #14 on:** July 19, 2012, 07:49:15 am »

Quote from: Alpha on July 17, 2012, 06:18:30 pm

Wait! No programming of the lexer necessary! Add:

AWESOME! Thank you Alpha, that works awesomely! I am at a loss why we need to deal with 3 different numbers (19,3,4) for indexing, but nevertheless, very nice

I don't suppose you know to change something like:

#define SOMEVAR(x) somefunc(x)
to just be
#define SOMEVAR(x) somefunc(x)

It seems to want to highlight the whole line.

Alpha · « **Reply #15 on:** July 19, 2012, 10:43:55 pm »

Here is the quick hack. (To do it properly, this should actually be connected to a checkbox in the settings.)

Code

Index: src/sdk/wxscintilla/src/scintilla/lexers/LexCPP.cxx
===================================================================
--- src/sdk/wxscintilla/src/scintilla/lexers/LexCPP.cxx	(revision 8148)
+++ src/sdk/wxscintilla/src/scintilla/lexers/LexCPP.cxx	(working copy)
@@ -221,7 +221,7 @@
 	bool highlightWxSmith;
 /* C::B end */
 	OptionsCPP() {
-		stylingWithinPreprocessor = false;
+		stylingWithinPreprocessor = true;
 		identifiersAllowDollars = true;
 		trackPreprocessor = true;
 		updatePreprocessor = true;

MortenMacFly · « **Reply #16 on:** July 20, 2012, 02:22:14 pm »

Quote from: Alpha on July 19, 2012, 10:43:55 pm

(To do it properly, this should actually be connected to a checkbox in the settings.)

Which are roughly 3 lines of code and a checkbox in a XRC file you can modify with C::B. Patches are welcome. (Not from you, Alpha - you've done enough already...)

Edit: See:
http://svn.berlios.de/wsvn/codeblocks/branches/xml_compiler/?op=revision&rev=8136&peg=8136
(Only the files editor_configuration.xrc, editorconfigurationdlg.cpp and cbeditor.cpp)

Code::Blocks Forums

News: