New syntax highlighting lexer (Motorola 68000 asm)

[Folco]

Hi,


I have used this page to write a lexer for my favorite language : http://wiki.codeblocks.org/index.php?title=Creating_a_custom_lexer_for_Code::Blocks_editor

The explanations help a lot, but there is still one thing that I don't understand. The full lexer is here : http://www.mirari.fr/6VIJ

What I don't understand it the index that I must put in the <Keywords> field :
<Set index="0" .../>, <Set index="1" .../> and <Set index="2" .../>


==> I don't understand to what "0", "1" and "2" refers to ?


As says the Wiki doc, I have downloaded Scite sources, looked at .properties files, but it didn't help me
In my case, the instructions are not in the right color and style (I wanted them bold).
Could someone give me some tip to me be able to complete this lexer ?

Thanx in advance.

[Folco]

Nobody can help me ?

I didn't find an already existing lexer. I didn't find a scintilla forum, I didn't find anything in the scintilla or scite doc, in the linux packages or in their sites...

If someone could show me some documentation, it would be very helpful !

[Folco]

I have succeed in doing a lexer which works quite fine ! http://www.mirari.fr/Nyp2

But there is still something I can't do : in asm 68k, hexa number are written $12AB, binaries %1010, integer are prefixed with '#' : #$ABCD.
I don't know how to say to the lexer that '$', '%' and '#' must be considered as numbers.

An idea ?


BTW, I have configured C::B to have tabs of eight chars, my .sample file contains tabs, but the preview uses 4-chars tabulations, not what the user have defined (it's a bug ??).
Is there a mean to change indentation size when using a specific syntax highlighting ?

I would also want to make some regular expressions to be considered as symbols or other things : [a-zA-Z]+[ @ ][a-zA-Z]. A mean to do that ?

[Folco]

Here a permanent link where you can download "Asm Motorola 68k" syntax highlighting : http://www.mirari.fr/fgP1

[Folco]

For those who would be interested in writing their own lexer, you can find some documentation in the sources of Scintilla shipped with Code::Blocks.
The SetIndex values are defined in a table at the end of the .cpp files which handle coloration. These files are in src/sdk/wxscintilla/src/scintilla/lexers/Lex*.cxx.

For the asm files, this table is used :

Code

static const char * const asmWordListDesc[] = {
    "CPU instructions",
    "FPU instructions",
    "Registers",
    "Directives",
    "Directive operands",
    "Extended instructions",
    0
};

The Index number (the "magic" number I was asking about in post #0) must follow this table :
0 for CPU Instructions
1 for FPU
2 for Registers
etc...

Quite simple, when you know where the informations are.

[Folco]

I have written a patch which handles better number for 68k assembly.
This patch modifies the file src/sdk/wxscintilla/src/scintilla/lexers/LexAsm.cxx

Additions :
'$' : hexa numbers
'%' : binary numbers
'#' : immediate values
'-' : negative
'~' : logical not

All the symbols can be used together, if the syntax is correct (incorrect numbers won't be colorized).

The archive with the lexer, the sample file, the patch and the readme are available here : http://www.mirari.fr/fgP1

[MortenMacFly]

This patch modifies the file src/sdk/wxscintilla/src/scintilla/lexers/LexAsm.cxx

As you've modified the scintilla core you should consider to send this patch to the scintilla developers (mailing list), too...?!

[Folco]

Morten, my patch is really ugly... I don't use MASM, so it's not a problem for me to break its highlighting, but this is not a clean way to propose it for upstream...

In fact, I really think that to do the job properly, I should add a totally new lexer, dedicated to 68k syntax.

I have listed the files I will have to change to perform that. Related files are :
wxscintilla/Lex68k.lo (file to create)
wxscintilla/Makefile (4 entries to modify)
wxscintilla/Makefile.am
wxscintilla/Makefile.in
wxscintilla/build/Makefile (1 entry to modify)
wxscintilla/build/wxscintilla.dsp
wxscintilla/scr/scintilla/lexers/Lex68k.cxx (file to create)
wxscintilla/include/ws/scintilla.h
What of these files are auto-generated, is there other things to modify ?
$(CB_src)/Makefile ? $(CB_src)/configure ?

I have also seen a wxscintilla/src/scintilla/lexers/.svn/ directory, which also contains lexer*.cxx and some other files... What about this hidden directory Some changes to do here ?


(Sorry for these questions, but I am not a real programer, just an amateur. I usually use a toolchain for 68k programming on embedded devices, I am not used to modify such big projects, and I don't use autotools...)


EDIT ->
wxscintilla/Lex*.lo -> seem to be auto-generates
wxscintilla/Makefile.am -> I think I must modify this
wxscintilla/Makefile.in -> autogenerated
wxscintilla/build/Makefile -> I think I must modify these files
wxscintilla/build/wxscintilla.dsp -> don't care about it, MS project file
wxscintilla/scr/scintilla/lexers/Lex68k.cxx -> write from scratch
wxscintilla/include/ws/scintilla.h -> add new entries

[Folco]

At the moment, I have done modifications in these files :

Code

Makefile.am          replace in    wxscintilla/
Makefile             replace in    wxscintilla/build/
wxscintilla.h        replace in    wxscintilla/include/wx/
Lex68k.cxx           add in        wxscintilla/src/scintilla/lexers/
lexer_68k.xml        add in        CBsrc/src/sdk/ressources/lexers
lexer_68l.sample     add in        CBsrc/src/sdk/ressources/lexers
Scintilla.iface      replace in    wxscintilla/src/scintilla/include
SciLexer.h           replace in    wxscintilla/src/scintilla/include

I don't know if there is other things to do.

[MortenMacFly]

EDIT ->
wxscintilla/Lex*.lo -> seem to be auto-generates
wxscintilla/Makefile.am -> I think I must modify this
wxscintilla/Makefile.in -> autogenerated
wxscintilla/build/Makefile -> I think I must modify these files
wxscintilla/build/wxscintilla.dsp -> don't care about it, MS project file
wxscintilla/scr/scintilla/lexers/Lex68k.cxx -> write from scratch
wxscintilla/include/ws/scintilla.h -> add new entries

Relevant are only:
wxscintilla/Makefile.am -> I think I must modify this
wxscintilla/scr/scintilla/lexers/Lex68k.cxx -> write from scratch
wxscintilla/include/ws/scintilla.h -> add new entries

...and the Code::Blocks project files if you add sources.

Ignore the others.

[Folco]

Thanks a lot ! \o/

[Folco]

Ok, I get the new lexer working. It doesn't break anymore Masm syntax highlighting.

But there were more files to modify :
- Scintilla.iface
- SciLexer.h
- Catalogue.cxx
I have still some tweaks to do to handler better the Motorola syntax, then I will post patches.

Morten, thanks again for your help.

[Folco]

Should be the "final release" : http://www.mirari.fr/G0Sn

I rewrote the parser. It now handles 15 different contexts.
This parser needs to modify 5 files of Scintilla.

I've tried to contact Scintilla dev, but I never used GoogleGroups before. Waiting for an answer. ^^

ps -> the main code is available here : http://www.yaronet.com/posts.php?sl=&h=133&s=135345&p=5#131 (just click the LexA68k.cxx box at the end of the post).

[Folco]

Please, does someone know how to contact Scintilla team ? I don't succeed to give them my patch :cry:
They don't seem to have a mail box, a forum or an irc chan...

I alse updated it, it know support Doxygen keywords ans "special words" (FIXME, DEBUG etc...).

[oBFusCATed]

Have you tried this: http://sourceforge.net/projects/scintilla/support

[ollydbg]

@Folco
from this page:
http://www.scintilla.org/ScintillaHistory.html
all the changes was list in the sourceforge, so I think you can add a feature request on the sourceforge.

[Folco]

Mainteners of Scintilla have commited my patch. First, thanks for your help.

But the guy said me to give patches for wxWidgets to the maitneners. I think the guy is talking about wxScintilla, no ? So I must give the patch to the C::B team ?

http://sourceforge.net/tracker/index.php?func=detail&aid=3101598&group_id=2439&atid=352439

[stahta01]

Mainteners of Scintilla have commited my patch. First, thanks for your help.

But the guy said me to give patches for wxWidgets to the maitneners. I think the guy is talking about wxScintilla, no ? So I must give the patch to the C::B team ?

http://sourceforge.net/tracker/index.php?func=detail&aid=3101598&group_id=2439&atid=352439

I agree, the patches to wxScintilla (that are not in Scintilla) needs to be submitted to Code::Blocks dev team.

But, I am just a Code::Blocks user/patch submitter myself.

Tim S.

[Folco]

Perhaps I should wait that the Team merge a recent version of Scintilla before proposing a patch ?
Indeed, the rank of the lexer is probably not the same in Scintilla that in the version shipped with C::B, or some other things like that ?

BTW, are recent Scintilla sources often merged in C::B ?

[oBFusCATed]

Yes, there is scintilla branch in the svn.
I think that the current version is 2.1 something...
You can search the svn log for more details.

[MortenMacFly]

I agree, the patches to wxScintilla (that are not in Scintilla) needs to be submitted to Code::Blocks dev team.

Don't worry, changes in scintilla will sooner or later make it into C::B. We keep scintilla quite up-to-date.

In fact I have some modifications pending, including the update to the latest release of scintilla (not wxScintilla).

Please understand that we won't apply development releases, this will have to wait at least until the next scintilla release.

[Folco]

No problem. I have a script which retrieves svn sources, patches them and build CB, so it's not a pain.

But you're talking about minor release ? The ones which are listed in this page ? http://www.scintilla.org/ScintillaHistory.html
As I can see, release cycle is short (about 2 monthes).

[MortenMacFly]

But you're talking about minor release ? The ones which are listed in this page ? http://www.scintilla.org/ScintillaHistory.html
As I can see, release cycle is short (about 2 monthes).

Not all of them, but when it makes sense - yes.