Perhaps making our own customized model would be easier. I mean, we can deduce the "C++ wannabe model" :lol: from Yiannis' code, and later we can expand it to other languages.
I looked at Yiannis' tokenizer (which would be better called 'parser', don't you love all that confusing nomenclature?

), and it has more or less a good model for deciding when to add a description for a function, keeps track of opening and closing brackets, etc.
Some of that functions can't be replicated (easily) with a standard FSA, so i'll add a new item to the table: "action". So instead of having only a "next state" given a state and an input, we'll have both an action, and a next state. That'll facilitate things.
EDIT: But it all depends on whether ANTLR has a generator, or it's more a generalized-parser model.