Code::Blocks Forums

User forums => Using Code::Blocks => Topic started by: Folco on October 15, 2010, 10:20:36 pm

Title: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 15, 2010, 10:20:36 pm
Hi,


I have used this page to write a lexer for my favorite language : http://wiki.codeblocks.org/index.php?title=Creating_a_custom_lexer_for_Code::Blocks_editor

The explanations help a lot, but there is still one thing that I don't understand. The full lexer is here : http://www.mirari.fr/6VIJ

What I don't understand it the index that I must put in the <Keywords> field :
<Set index="0" .../>, <Set index="1" .../> and <Set index="2" .../>


==> I don't understand to what "0", "1" and "2" refers to ?


As says the Wiki doc, I have downloaded Scite sources, looked at .properties files, but it didn't help me :(
In my case, the instructions are not in the right color and style (I wanted them bold).
Could someone give me some tip to me be able to complete this lexer ?

Thanx in advance. :)

(http://www.mirari.fr/aKZB) (http://www.mirari.fr/XId7)
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 17, 2010, 01:31:16 pm
Nobody can help me ?

I didn't find an already existing lexer. I didn't find a scintilla forum, I didn't find anything in the scintilla or scite doc, in the linux packages or in their sites...

If someone could show me some documentation, it would be very helpful !
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 17, 2010, 08:47:58 pm
I have succeed in doing a lexer which works quite fine ! http://www.mirari.fr/Nyp2

But there is still something I can't do : in asm 68k, hexa number are written $12AB, binaries %1010, integer are prefixed with '#' : #$ABCD.
I don't know how to say to the lexer that '$', '%' and '#' must be considered as numbers.

An idea ?


BTW, I have configured C::B to have tabs of eight chars, my .sample file contains tabs, but the preview uses 4-chars tabulations, not what the user have defined (it's a bug ??).
Is there a mean to change indentation size when using a specific syntax highlighting ?

I would also want to make some regular expressions to be considered as symbols or other things : [a-zA-Z]+[ @ ][a-zA-Z]. A mean to do that ?
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 17, 2010, 09:12:01 pm
Here a permanent link where you can download "Asm Motorola 68k" syntax highlighting : http://www.mirari.fr/fgP1
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 18, 2010, 05:36:59 pm
For those who would be interested in writing their own lexer, you can find some documentation in the sources of Scintilla shipped with Code::Blocks.
The SetIndex values are defined in a table at the end of the .cpp files which handle coloration. These files are in src/sdk/wxscintilla/src/scintilla/lexers/Lex*.cxx.

For the asm files, this table is used :
Code
static const char * const asmWordListDesc[] = {
    "CPU instructions",
    "FPU instructions",
    "Registers",
    "Directives",
    "Directive operands",
    "Extended instructions",
    0
};
The Index number (the "magic" number I was asking about in post #0) must follow this table :
0 for CPU Instructions
1 for FPU
2 for Registers
etc...

Quite simple, when you know where the informations are. :D
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 18, 2010, 10:33:45 pm
I have written a patch which handles better number for 68k assembly.
This patch modifies the file src/sdk/wxscintilla/src/scintilla/lexers/LexAsm.cxx

Additions :
'$' : hexa numbers
'%' : binary numbers
'#' : immediate values
'-' : negative
'~' : logical not

All the symbols can be used together, if the syntax is correct (incorrect numbers won't be colorized).

The archive with the lexer, the sample file, the patch and the readme are available here : http://www.mirari.fr/fgP1
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: MortenMacFly on October 19, 2010, 07:05:06 am
Quote from: Folco on October 18, 2010, 10:33:45 pm
This patch modifies the file src/sdk/wxscintilla/src/scintilla/lexers/LexAsm.cxx
As you've modified the scintilla core you should consider to send this patch to the scintilla developers (mailing list), too...?!
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 19, 2010, 11:45:57 am
Morten, my patch is really ugly... I don't use MASM, so it's not a problem for me to break its highlighting, but this is not a clean way to propose it for upstream...

In fact, I really think that to do the job properly, I should add a totally new lexer, dedicated to 68k syntax.

I have listed the files I will have to change to perform that. Related files are :
wxscintilla/Lex68k.lo (file to create)
wxscintilla/Makefile (4 entries to modify)
wxscintilla/Makefile.am
wxscintilla/Makefile.in
wxscintilla/build/Makefile (1 entry to modify)
wxscintilla/build/wxscintilla.dsp
wxscintilla/scr/scintilla/lexers/Lex68k.cxx (file to create)
wxscintilla/include/ws/scintilla.h
What of these files are auto-generated, is there other things to modify ?
$(CB_src)/Makefile ? $(CB_src)/configure ?

I have also seen a wxscintilla/src/scintilla/lexers/.svn/ directory, which also contains lexer*.cxx and some other files... What about this hidden directory ??? Some changes to do here ?


(Sorry for these questions, but I am not a real programer, just an amateur. I usually use a toolchain for 68k programming on embedded devices, I am not used to modify such big projects, and I don't use autotools...)


EDIT ->
wxscintilla/Lex*.lo -> seem to be auto-generates
wxscintilla/Makefile.am -> I think I must modify this
wxscintilla/Makefile.in -> autogenerated
wxscintilla/build/Makefile -> I think I must modify these files
wxscintilla/build/wxscintilla.dsp -> don't care about it, MS project file
wxscintilla/scr/scintilla/lexers/Lex68k.cxx -> write from scratch
wxscintilla/include/ws/scintilla.h -> add new entries
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 19, 2010, 01:29:40 pm
At the moment, I have done modifications in these files :
Code
Makefile.am          replace in    wxscintilla/
Makefile             replace in    wxscintilla/build/
wxscintilla.h        replace in    wxscintilla/include/wx/
Lex68k.cxx           add in        wxscintilla/src/scintilla/lexers/
lexer_68k.xml        add in        CBsrc/src/sdk/ressources/lexers
lexer_68l.sample     add in        CBsrc/src/sdk/ressources/lexers
Scintilla.iface      replace in    wxscintilla/src/scintilla/include
SciLexer.h           replace in    wxscintilla/src/scintilla/include
I don't know if there is other things to do.
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: MortenMacFly on October 19, 2010, 03:44:21 pm
Quote from: Folco on October 19, 2010, 11:45:57 am
EDIT ->
wxscintilla/Lex*.lo -> seem to be auto-generates
wxscintilla/Makefile.am -> I think I must modify this
wxscintilla/Makefile.in -> autogenerated
wxscintilla/build/Makefile -> I think I must modify these files
wxscintilla/build/wxscintilla.dsp -> don't care about it, MS project file
wxscintilla/scr/scintilla/lexers/Lex68k.cxx -> write from scratch
wxscintilla/include/ws/scintilla.h -> add new entries
Relevant are only:
wxscintilla/Makefile.am -> I think I must modify this
wxscintilla/scr/scintilla/lexers/Lex68k.cxx -> write from scratch
wxscintilla/include/ws/scintilla.h -> add new entries

...and the Code::Blocks project files if you add sources.

Ignore the others.
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 19, 2010, 03:55:51 pm
Thanks a lot ! \o/
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 19, 2010, 06:43:34 pm
Ok, I get the new lexer working. It doesn't break anymore Masm syntax highlighting.

But there were more files to modify :
- Scintilla.iface
- SciLexer.h
- Catalogue.cxx
I have still some tweaks to do to handler better the Motorola syntax, then I will post patches.

Morten, thanks again for your help. :)
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on October 22, 2010, 01:07:01 am
Should be the "final release" : http://www.mirari.fr/G0Sn

I rewrote the parser. It now handles 15 different contexts.
This parser needs to modify 5 files of Scintilla.

I've tried to contact Scintilla dev, but I never used GoogleGroups before. Waiting for an answer. ^^

ps -> the main code is available here : http://www.yaronet.com/posts.php?sl=&h=133&s=135345&p=5#131 (just click the LexA68k.cxx box at the end of the post).
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on November 02, 2010, 10:40:49 am
Please, does someone know how to contact Scintilla team ? I don't succeed to give them my patch :cry:
They don't seem to have a mail box, a forum or an irc chan...

I alse updated it, it know support Doxygen keywords ans "special words" (FIXME, DEBUG etc...).
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: oBFusCATed on November 02, 2010, 10:48:18 am
Have you tried this: http://sourceforge.net/projects/scintilla/support
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: ollydbg on November 02, 2010, 12:05:09 pm
@Folco
from this page:
http://www.scintilla.org/ScintillaHistory.html
all the changes was list in the sourceforge, so I think you can add a feature request on the sourceforge.
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on November 06, 2010, 12:12:44 am
Mainteners of Scintilla have commited my patch. First, thanks for your help. :)

But the guy said me to give patches for wxWidgets to the maitneners. I think the guy is talking about wxScintilla, no ? So I must give the patch to the C::B team ?

http://sourceforge.net/tracker/index.php?func=detail&aid=3101598&group_id=2439&atid=352439
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: stahta01 on November 06, 2010, 01:20:09 am
Quote from: Folco on November 06, 2010, 12:12:44 am
Mainteners of Scintilla have commited my patch. First, thanks for your help. :)

But the guy said me to give patches for wxWidgets to the maitneners. I think the guy is talking about wxScintilla, no ? So I must give the patch to the C::B team ?

http://sourceforge.net/tracker/index.php?func=detail&aid=3101598&group_id=2439&atid=352439

I agree, the patches to wxScintilla (that are not in Scintilla) needs to be submitted to Code::Blocks dev team.

But, I am just a Code::Blocks user/patch submitter myself.

Tim S.
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on November 06, 2010, 12:33:46 pm
Perhaps I should wait that the Team merge a recent version of Scintilla before proposing a patch ?
Indeed, the rank of the lexer is probably not the same in Scintilla that in the version shipped with C::B, or some other things like that ?

BTW, are recent Scintilla sources often merged in C::B ?
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: oBFusCATed on November 06, 2010, 01:00:15 pm
Yes, there is scintilla branch in the svn.
I think that the current version is 2.1 something...
You can search the svn log for more details.
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: MortenMacFly on November 06, 2010, 09:25:32 pm
Quote from: stahta01 on November 06, 2010, 01:20:09 am
I agree, the patches to wxScintilla (that are not in Scintilla) needs to be submitted to Code::Blocks dev team.
Don't worry, changes in scintilla will sooner or later make it into C::B. We keep scintilla quite up-to-date.

In fact I have some modifications pending, including the update to the latest release of scintilla (not wxScintilla).

Please understand that we won't apply development releases, this will have to wait at least until the next scintilla release.
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: Folco on November 06, 2010, 10:24:26 pm
No problem. I have a script which retrieves svn sources, patches them and build CB, so it's not a pain.

But you're talking about minor release ? The ones which are listed in this page ? http://www.scintilla.org/ScintillaHistory.html
As I can see, release cycle is short (about 2 monthes).
Title: Re: New syntax highlighting lexer (Motorola 68000 asm)
Post by: MortenMacFly on November 08, 2010, 06:40:27 am
Quote from: Folco on November 06, 2010, 10:24:26 pm
But you're talking about minor release ? The ones which are listed in this page ? http://www.scintilla.org/ScintillaHistory.html
As I can see, release cycle is short (about 2 monthes).
Not all of them, but when it makes sense - yes.