As we know, Quex lexer generate some code directed lexer which is much faster then the "table driving lexer" like flex. see:
Lexical analysis - Wikipedia, the free encyclopediaQuex - A Fast Universal Lexical Analyzer Generatorand the performance compared with flex
SourceForge.net: Lexical Analyzer Generator Quex: Topic: Performance question about Quex post 5
And I just create a lexer grammar here
SourceForge.net: Lexical Analyzer Generator Quex: Modify: 3074664 - a c++ quex lexer codeBy using a lexer generator, we can remove a lot of hand-craft codes in our currently Tokenizer class, and I hope it will let our CC much faster.
I have a simple demo test for Parserthread using the "TokenID and text infromation" from this lexer, This way, we can avoid a lot of wxString comparing statement in Parserthread's code, instead, we used TokenID comparison, which is only
int type comparison or a table driven jump can be used.
but the demo test is quite simple, I hope I can improved it in the future.