Author Topic: New parser model for Code completion  (Read 55035 times)

Offline killerbot

  • Administrator
  • Lives here!
  • *****
  • Posts: 5177
Re: New parser model for Code completion
« Reply #30 on: December 12, 2005, 09:14:47 pm »
Hello parser experts,

Maybe you can shed some light on this bug report of mine :
https://sourceforge.net/tracker/index.php?func=detail&aid=1323191&group_id=126998&atid=707416

The code completion does not kick in on function arguments. Is this a shortcoming in the current strategy/parser, or is it possible with the current mechanism and are we just suffering from a ?minor/major? bug. Any ideas for fixes ?

Thanks for your time,
Lieven

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: New parser model for Code completion
« Reply #31 on: December 12, 2005, 09:20:50 pm »
Rick, I want to say that I encourage your own parser work, but it is certain that you'll have to spend some years to reach a state compared to the other works.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: New parser model for Code completion
« Reply #32 on: December 13, 2005, 05:44:44 am »
TakeshiMiya: As I said, I only want to replicate Yiannis' parser functionality. Then we might be able to extend it or perhaps use the other parsers.

In any case, I'm still working on my tree model to improve the current parser's speed. I don't think it'll be easy to combine the current parser's memory model with the other parsers. In that case it would be better to start them completely from scratch, using a new memory model adapted to fit them. But I don't think I'm qualified to do that, because I don't know those parsers. On the other hand, you do seem to have knowledge about them :-)

(Besides, I've always wanted to do this, it's something like a personal challenge. Just like byo wanted to write his own RAD editor, I want to write my parser. But I don't guarantee that i might succeed on time, or even have be able to START working on it. As I said, my current circumstances only allow me to work on C::B half an hour daily, even less. So if anyone wants to try making their own improved code completion based on these parsers, they're welcome. Since my approach is different, I don't think I'll interfere with it.

Well, it's late now and I have to go to bed. See ya.

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: New parser model for Code completion
« Reply #33 on: December 13, 2005, 11:49:47 am »
Hello Rick and parser experts,

In a book I have bought time last year in Bangkok (I know that it would be a waste of money :)), there is a "simple" implementation of a parser for C++. This parser for C++ is part of a Mini-Interpreter for C++.

The implementation of the parser looks interesting and may be it could be useful for the development of your parser.

The code of the Mini-Interpreter for C++ belongs to the book The Art of C++ and It is freely available here.

After unzipping the file, use a text editor (As UltraEdit) to get the files of the Mini-Interpreter C++ (it is CHAP9.LST).

Best wishes,
Michael
« Last Edit: March 01, 2006, 11:23:30 am by Michael »

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: New parser model for Code completion
« Reply #34 on: December 20, 2005, 07:26:20 pm »
Guys, look at this!

http://www.cee.hw.ac.uk/~alison/alg/lectures.html

I think these lectures will benefit us all.

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: New parser model for Code completion
« Reply #35 on: December 20, 2005, 07:34:48 pm »
Hmm... that reads almost like Sedgewick's book :)


Oh... lol
Quote
Sedgewick, chapters 19-23
« Last Edit: December 20, 2005, 07:37:00 pm by thomas »
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: New parser model for Code completion
« Reply #36 on: December 20, 2005, 07:42:33 pm »
Guys, look at this!

http://www.cee.hw.ac.uk/~alison/alg/lectures.html

I think these lectures will benefit us all.

Interesting lectures, even if a bit old :). Some external links are unfortunately broken :(.

Michael

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: New parser model for Code completion
« Reply #37 on: December 21, 2005, 12:54:55 am »
Wouldn't the ideal system do:

1.  use lexx to generate a lexer/tokenizer/fsm that would be run to tokenize the source files into tokens.
2.  yacc would then, given language tokens, parse them into valid language syntax and generate an abstract syntax tree (each node is a token - organized in a tree structure to reflect the language grammer - i.e. (x+5) +3 != x + (5 + 3) even though they have the same tokens - x, 5, 3).

So to add a different parser all that would need to be done is:
a) specify the lexx input file and run lexx on it - this produces a fsm
b) use yacc on a grammar specification based on the tokens lexx produces to produce the parser.

The parser will then parse the language specified and give you an abstract syntax tree to work with.  These tree nodes can be annotated with things like comments (although then it has to be agreed that comments must preced source lines unless they are on the same line or ...) or other attributes.

The tree will then be what codeblocks uses to iterate over to get the info it needs for code completion, find declaration/implementation, refactoring?, ...

CB wouldn't need to be recompiled to add a different language - just lexx & yacc run on the specification, then maybe an .xml config file edited which CB uses to determine which languages are available.

Isn't this the most robust & simplest solution to get up & running?

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: New parser model for Code completion
« Reply #38 on: December 21, 2005, 01:10:32 am »
Isn't this the most robust & simplest solution to get up & running?

No, the most one I found is CodeStore, which in fact uses ANTLR C++ (which is a perfect replace for lexx/yacc/bison/etc)

If we use ANTLR, we get automatically these grammars done by the community:
-Full and mature, C++ supporting almost everything (templates, namespaces, etc)
-Phyton
-C#
-Java
-Pascal
-MySQL
-HTML
-CSS
-JavaScript
-Ada
-Verilog
-A lot more

And being that is a parser generator, it's intended to add any other language you want.

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: New parser model for Code completion
« Reply #39 on: December 21, 2005, 01:34:29 am »
So what's the issue with integrating antlr?  It sounds like the java ant makefile compiler mixed with LR parsing...

I've used JavaCC with JCUP before for writing a compiler so I'd be willing to help integrate if needed.
So the output of antlr is a c++ parser or...?

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: New parser model for Code completion
« Reply #40 on: December 21, 2005, 01:48:08 am »
So the output of antlr is a c++ parser or...?
Yes, and it's better even, because the ANTLR output can be: a C++ parser, a Python parser, a C# parser, or a Java parser.
Im unsure if the output based on the grammars is written inside ANTLR, or it depends on the grammar developer, but for sure the C++ parser output is C++.

From what I've read ANTLR is like JavaCC but a lot better.

Anyways, take a look at CodeStore, that it's the work of another opensource IDE, which are some classes around the ANTLR C++ parser to facilitate the job (dealing with the AST, etc), with the purpose of code completion.

It's very new so you'll have to get it from CVS.
I've tried compiling it (CodeStore with ANTLR C++) in Code::Blocks and compiles almost out-of-the-box, it's all written in portable C++. :)

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: New parser model for Code completion
« Reply #41 on: December 21, 2005, 02:55:23 am »
If we're talking about VCFBuilder then I don't think the CVS source is complete.  There's includes like:

#   include "../src/CodeStore/src/CodeStore.h"

That aren't in the source tree.  Uses doxygen which is nice, but I can't seem to get it to compile or see the internals of the codesource source.

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: New parser model for Code completion
« Reply #42 on: December 21, 2005, 03:16:36 am »
Mm... There are various branches, I don't remember what I used.

Basically I imported the MSVC6 project to C::B.
Then I changed some classes assignations because a bug that GCC 3.4 haves in the ANTLR code, and it compiled without errors.

And for the CodeStore part, I remember changing some things to make it work, I got it working too.
I don't know what is the level of completeness of CodeStore, but surelly is worth the look, after all it's very new and done in portable C++.
The best would be join the development of the CodeStore branch and help in the development.

Offline Game_Ender

  • Lives here!
  • ****
  • Posts: 551
Re: New parser model for Code completion
« Reply #43 on: December 21, 2005, 04:34:33 am »
How does the CodeStore parser compare to Elsa?  I have been looking and looking for a good, feature complete, and tested C++ parser library and Elsa is it.  It even has the same license as CodeStore, and it looks more complete.  It would definitely not be a general solution but the again, from everything I have been reading you can't parse C++ like you can most other languages, so having a separate method of doing the C++ parsing/AST building might not be a bad thing.

Offline anonuser

  • Multiple posting newcomer
  • *
  • Posts: 44
Re: New parser model for Code completion
« Reply #44 on: December 21, 2005, 04:37:11 am »
Alright so why don't we just take a stab at making the plugin?