Author Topic: Syntax Highlighting -- Is it really this hard?  (Read 15664 times)

Offline marphod

  • Single posting newcomer
  • *
  • Posts: 7
Syntax Highlighting -- Is it really this hard?
« on: May 10, 2013, 06:37:01 am »
Having looked, and not found, a comprehensive syntax highlighting theme for Code::Blocks, I decided to take a look at what would be involved in making one.

As far as I can tell, there is no 'simple' way to make a comprehensive theme.  Either you go through the highlighting options item-for-item through the GUI, or you hand craft one by duplicating the highlight style sections for each lexer, dealing with each lexer separately.

There doesn't seem to bey a way to say 'This is the default background and foreground color, for ALL languages', 'This is the background and foreground color for all comments for all languages', etc.  Each language has between 10 and 50 options, and each language has to be dealt with on its own.

(As it stands, the default theme is annoyingly inconsistent, but in some cases only by a little bit.  For instance, comments are mostly fg #A0A0A0, except when they are #808080 or (...)  #7F7F7F. Although there are a few outliers.)


Am I missing something?


And if it really is that hard, has anyone written up a script to make life easier?  It shouldn't be that hard to write a perl (or python, or ruby, or whatever) script to slurp in the xml for each lexer and output a new one with a different theme. 

It'd require some field-name matching, but that shouldn't be too hard.  There should be about 50 basic field types and about 10 basic modifiers.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12122
    • Travis build status
Re: Syntax Highlighting -- Is it really this hard?
« Reply #1 on: May 10, 2013, 09:25:12 am »
It'd require some field-name matching, but that shouldn't be too hard.  There should be about 50 basic field types and about 10 basic modifiers.
Unfortunately it is not that simple. Look at the html/javascript/php lexer and you'll know why.
BTW: The latest nightly should have some gui, which should ease the setting of colours.
It allows you to copy colours from on language to another.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline marphod

  • Single posting newcomer
  • *
  • Posts: 7
Re: Syntax Highlighting -- Is it really this hard?
« Reply #2 on: May 11, 2013, 12:45:07 am »
It'd require some field-name matching, but that shouldn't be too hard.  There should be about 50 basic field types and about 10 basic modifiers.
Unfortunately it is not that simple. Look at the html/javascript/php lexer and you'll know why.

Unless I'm missing something, it really is.

<Embedded language> <foo> would map to the <Embedded language> Background and the <foo> foreground (Comment, Default, Number, whatever).  If <foo> has a different background from default, do some color blending of it and the embed backrgound.

You'll need a embedded background for (IIRC) Python, Javascript, SGML, VBS, ASM (for C/C++),  PHP, and the ASP versions of Python, VBS, and javascript.


It means doing some string parsing on top of simple string matches and an alias table.






Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12122
    • Travis build status
Re: Syntax Highlighting -- Is it really this hard?
« Reply #3 on: May 11, 2013, 02:28:36 am »
Unless I'm missing something, it really is.
Patches welcome, but be prepared for bug fixing, I'll be happy to find lots of them  ;D
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline marphod

  • Single posting newcomer
  • *
  • Posts: 7
Re: Syntax Highlighting -- Is it really this hard?
« Reply #4 on: May 13, 2013, 11:14:40 pm »
Patches welcome, but be prepared for bug fixing, I'll be happy to find lots of them  ;D

Fair enough. (I have always hated QA ;) (1) )

Although, my prototype is a stand-alone text-based perl script. 
I'll post a link (or the code, if it remains short) when it is
at a working stage.




(1) - JOKE.  I love QA, I just couldn't do it, professionally, myself(2).
(2) - Which is to say while I totally believe in unit tests, test-driven development, and rapid test cycles, I want to fix things rather than just note what's broken.

Offline marphod

  • Single posting newcomer
  • *
  • Posts: 7
Gaaaah. The lexer XML is not compliant
« Reply #5 on: May 15, 2013, 05:28:18 pm »
I think I would be code-complete on my proof of concept, EXCEPT I've made the fun discovery that the various Lexer XML files' XML is non-complaint in a number of ways.

(Unquoted attributes, using a raw Ampersand and Open angle brace in strings, and possibly other issues.)

Blah.  Off to find a  more permissive Perl XML parser than LibXML. 


Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12122
    • Travis build status
Re: Gaaaah. The lexer XML is not compliant
« Reply #6 on: May 15, 2013, 06:20:49 pm »
(Unquoted attributes, using a raw Ampersand and Open angle brace in strings, and possibly other issues.)
Why don't you fix them and provide a patch?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline marphod

  • Single posting newcomer
  • *
  • Posts: 7
Re: Gaaaah. The lexer XML is not compliant
« Reply #7 on: May 16, 2013, 06:12:55 am »
Why don't you fix them and provide a patch?

I've no idea where in the source tree I'd find them.
For that matter, are they a Code::Blocks thing, or are they sourced from Scintilla?
(and I've had issues logging on to http://developer.berlios.de -- it keeps telling me to turn cookies on, but I've allowed them for their domain, and quite a few others.)
(Heck, I really hope they aren't hand-generated, but programmatically created.  But probably not.)

- - -

Regardless: Why submit a patch of a few XML files when I have a working prototype, which can fix the XML automagically.  =)
(Well, except for the weird spacing in attributes; I maintained that, as it helps readability and the fix may be beyond the scope of the XML.)

(If you want to do so, comment out the line "SetThemeForNode( $theme_map, $style_node, @theme_names );"; line 423.  It will regenerate the .xml files in a more
standard's compliant format.)



This is a PROTOTYPE. 
It creates new versions of the lexer_<foo>.xml files with the new syntax highlighting styles. 

The program works buy slurping in the existing lexer files and regenerating them (in another location) with the new style settings. 
You need to replace the .xml files (/usr/share/codeblocks/lexer/*.xml on fedora 18, and probably most other unix/linux installs).  BACK THE ORIGINALS UP FIRST.

You have to set the theme settings (colors, bold, etc.) by hand in the cb_theme.xml file (sample attached); well, assuming you don't want to use my semi-random/semi-use-based settings.  You will also  probably disagree with some of the name/alias mappings I made in the cb_theme.xml files.  That's fine, the XML file is fairly straight-forward. 

There are known issues (the program will abort if it finds the same theme-type name/alt-name multiple times in the config file.  The program will abort if it finds a style name in a lexer file it does not recognize, rather than treating it as 'no style attributes'.  The Pretty-print output has occasional issues (some nodes being more indented than they should; some less; most attributes are one-per-line, but not all). 

Cavaet Emptor.

- - -

(Future versions will create themes that can be imported, rather than change the theme file.  This is a proof of concept prototype.  Future versions will also have README, CHANGES, etc., rather than just the perl code and sample config file.)

Requires:
libxml2
perl 5.014
Perl Modules:
Getopt::Long::Descriptive
XML::LibXML



Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1513
Re: Gaaaah. The lexer XML is not compliant
« Reply #8 on: May 16, 2013, 04:54:32 pm »
I've no idea where in the source tree I'd find them.
For that matter, are they a Code::Blocks thing, or are they sourced from Scintilla?
They are Code::Blocks specific, located in src/sdk/resources/lexers/ (if you check out the source tree).  There is also some information about the purpose of these files.

Offline marphod

  • Single posting newcomer
  • *
  • Posts: 7
Re: Gaaaah. The lexer XML is not compliant
« Reply #9 on: May 22, 2013, 11:34:42 pm »
They are Code::Blocks specific, located in src/sdk/resources/lexers/ (if you check out the source tree).  There is also some information about the purpose of these files.

Noted.  I'll take a look at them as soon as I manage to check out the source tree.

(. . . So which of the various SCM plugins is the most useable at this point? )

- - -

Also, for the ... 2 of you who downloaded the prototype, here's the next version.

The only important change to the command line is there are now --lexerstyle and --configstyle flags (defaults to configstyle), which controls the type of output generated.  lexerstyle is the same as the previous version (new versions of the lexer files, which is more robust against accidental changes, but requires root to install, as I understand it).  configstyle creates an xml (.conf) file with various colour_style settings for each language. 

--configstyle also can take an optional string, to be the name of the new theme and will be used in the created file name; it defaults to newTheme (so the file will be newTheme.conf)

(These flags are mutually exclusive, although the error message from Getopts::Long::Descriptive isn't terribly helpful in explaining that.)


I'm probably going to put futher feature work on this project on indefinate hiatus, at least until/unless I get some actual commentary and feedback.  It isn't perfect, but it works for my needs.  I'll try to respond to bug reports, and I plan on putting it up on sourceforge or github in the near future. (I'll add a link here when that's done.)

- - -


Attached file contents:
LICENSE
README
TODO
CHANGES
createCBTheme.pl
cb-theme.xml

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12122
    • Travis build status
Re: Gaaaah. The lexer XML is not compliant
« Reply #10 on: May 22, 2013, 11:53:53 pm »
(. . . So which of the various SCM plugins is the most useable at this point? )
I think it was called terminal :)
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]