Author Topic: New parser model for Code completion  (Read 56395 times)

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: New parser model for Code completion
« Reply #60 on: December 21, 2005, 07:21:32 am »
Yeah, that's great! 0 Java dependency 8)

And if we use parsers generated by ANTLR like the C++ one, we don't even need the parser generator because the parser is already generated (until you want to modify the parser generator itself).

So putting it easy: C++ for the parser generator, C++ for the parser.

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: New parser model for Code completion
« Reply #61 on: December 21, 2005, 02:08:30 pm »
Hello,

please allow me to put a link to topic where I have posted about the expression parser of the Mini C++. It is here.

Michael

Offline anonuser

  • Multiple posting newcomer
  • *
  • Posts: 44
Re: New parser model for Code completion
« Reply #62 on: December 21, 2005, 04:37:46 pm »
just but what about us unix folks? I'm all for a no java depnds. but there is a library we can link to instead.

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: New parser model for Code completion
« Reply #63 on: December 21, 2005, 06:08:13 pm »
unix compatibility should be a nonissue - the generated parsing code is portable.

Offline anonuser

  • Multiple posting newcomer
  • *
  • Posts: 44
Re: New parser model for Code completion
« Reply #64 on: December 21, 2005, 06:21:28 pm »
Well I work on unix and port back to other platforms. So I'll probably end up using Antlr C++

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: New parser model for Code completion
« Reply #65 on: December 23, 2005, 06:46:21 pm »
Any progress?  Getting the cpp parser to compile is a nightmare on mingw and even getting the antlr lib to compile on mingw is a mess.  Using the autoconf stuff doesn't seem to work ("no rule to build Makefile.in", among other errors) and well, I've pretty much given up on antlr2.7.6 lib / antlrcpp / antlr cpp parser combinations.  Maybe the codestore stuff is better in terms of working compiling instructions, etc.  Didn't bother to download vcl yet to try it out, but is the project even active (looked like stuff from 2 years ago iirc)?  Not much luck getting a simple working demonstration of this stuff on windows using mingw gcc.  Anyone mind posting detailed instructions on setup stuff if they've had sucess with getting something working?

Offline TDragon

  • Lives here!
  • ****
  • Posts: 936
    • Twilight Dragon Media
Re: New parser model for Code completion
« Reply #66 on: December 23, 2005, 07:34:49 pm »
I've been working with ANTLR 2.7.5 (I haven't gotten around to building 2.7.6 yet) and CPP_grammarV3.1 from the ANTLR website, with some good success. Out-of-the-box, it'll correctly parse a GCC-preprocessed file if it doesn't use any GCC extensions or built-in types (i.e. didn't include any standard headers); most of my time has been spent getting to know CPP_parser.g to figure out where and how I should add them. After adding support for most gcc __attribute__ specifiers and adding __builtin_va_list to the basic types, I came VERY close to successfully parsing Quadratic.cpp which includes two standard headers. Of the few remaining errors, most are related to the lack of support for "using" declarations when doing AST resolution lookup, so right now I've moved back to building a custom AST container which I'll drop in before I correct them.

After reading all the installation and usage instructions, only one step wasn't completely obvious, which was that I had to rebuild libantlr.a rather than using the pre-packaged version. Basically,
- Make sure you have MinGW and MSys installed (MSys is only needed to build the antlr library; it's fully MinGW32 compatible after this)
- Download and unzip antlr-x.y.z.tar.gz
- Open MSys, go to the antlr directory, run "./configure --disable-examples" (I added "--prefix=/path/to/antlr-dir" because I wasn't sure where the default was)
- Run "make install"
- If you want to be able to run antlr from the Windows command prompt, also download antlr-x.y.z.exe
- Download and unzip CPP_parserV3.1.zip
- If you want extended trace functionality, copy LLkParser.hpp to the antlr include/antlr directory; otherwise, comment out all references to antlrTrace() in CPP_parser.g
- Run antlr on CPP_parser.g to generate CPPParser.(cpp,hpp), CPPLexer.(cpp,hpp), and STDCTokenTypes.(hpp,txt)
- Compile CPPParser.cpp, CPPLexer.cpp, Dictionary.cpp, LLkParser.cpp (if you want extended trace), and Support.cpp into whatever project you want to use the parser in
- For the test project included with the parser, also compile Main.cpp and MyCode.cpp with MYCODE #defined

That was all from memory, but any problems you run into should be trivial. If you compile the test project, you can run the resulting executable on any preprocessed code (use gcc -E) to get a list of defined functions and optionally a list of declarations.

Hope that helps,
Twilight Dragon
« Last Edit: December 23, 2005, 07:38:33 pm by TDragon »
http://tdm-gcc.tdragon.net/ - TDM-GCC compiler suite for Windows (GCC 5.1.0 2015-06-28, 32/64-bit, no extra DLLs)

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: New parser model for Code completion
« Reply #67 on: December 23, 2005, 07:52:52 pm »
The project is actively mantained (last release was from 2 months ago or so).

I've used the MSVC6 projects of ANTLRC++ that comes with CodeStore, imported them and only had to change a class instantation which was a bug in MinGW. Compiled with 0,0 warnings/errors in C::B.

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: New parser model for Code completion
« Reply #68 on: December 23, 2005, 11:33:56 pm »
The project is actively mantained (last release was from 2 months ago or so).

I've used the MSVC6 projects of ANTLRC++ that comes with CodeStore, imported them and only had to change a class instantation which was a bug in MinGW. Compiled with 0,0 warnings/errors in C::B.

What's the link to the codestore download you're using?

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: New parser model for Code completion
« Reply #69 on: December 23, 2005, 11:40:31 pm »
No link, I fetched from CVS.  :)

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: New parser model for Code completion
« Reply #70 on: December 24, 2005, 12:51:55 am »
- Open MSys, go to the antlr directory, run "./configure --disable-examples" (I added "--prefix=/path/to/antlr-dir" because I wasn't sure where the default was)
- Run "make install"

See right here is where I get errors about not being able to build Makefile.in.  Could you post your `echo $PATH` in msys?

Offline anonuser

  • Multiple posting newcomer
  • *
  • Posts: 44
Re: New parser model for Code completion
« Reply #71 on: December 24, 2005, 01:54:40 am »
Alright I've got me antlr going and a working C++ grammer file.
Now just need to work with the AST it gives me.

Offline TDragon

  • Lives here!
  • ****
  • Posts: 936
    • Twilight Dragon Media
Re: New parser model for Code completion
« Reply #72 on: December 24, 2005, 04:03:59 am »
$ echo $PATH
.:/usr/local/bin:/mingw/bin:/bin:/mingw/bin:/e/PHP:/e/WINDOWS/system32:/e/WINDOWS:/e/WINDOWS/System32/Wbem:/e/Program Files/ATI Technologies/ATI Control Panel:.:/e/Program Files/doxygen/bin:/e/Python24:/e/Program Files/CVSNT/
http://tdm-gcc.tdragon.net/ - TDM-GCC compiler suite for Windows (GCC 5.1.0 2015-06-28, 32/64-bit, no extra DLLs)

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: New parser model for Code completion
« Reply #73 on: December 26, 2005, 09:25:46 pm »
Thanks TDragon.  It turned out to be something wrong with my msys configuration.  Reinstalling mingw and msys fixed things.  So getting antlr running wasn't so bad on mingw after all.  I didn't just use the supplied Main.cpp because I wanted to see how easy it would be to generalize that part for other parsers (as long as there is a .g file for the parser definition).  Tried to use sa few of the C++ specific extensions to antlr which were written as possible (e.g. no modding the antlr source to support his tracing extensions).  So it looks like modifying antlr.2.76 is not necessary at all - just build it first and then the c++ grammer does plug in fairly nicely.  Steps to get the parser working:

Code: [Select]
install mingw & msys:

binutils-2.16.91-20050827-1.tar.gz
extra.zip
gcc-core-3.4.4-20050522-1.tar.gz
gcc-g++-3.4.4-20050522-1.tar.gz
gdb-6.3-2.exe
mingw-runtime-3.9.tar.gz
mingw-utils-0.3.tar.gz
mingw32-make-3.80.0-3.tar.gz
MSYS-1.0.10.exe
w32api-3.5.tar.gz

extract all the archives to C:\MinGW, run the gdb installer, run the msys installer (use C:\MinGW for the postinstall prompt)

extract CPP_parserV3.1
copy CPP_parser.g, CPPDictionary.hpp, Dictionary.hpp, DictEntry.hpp, Dictionary.cpp, CPPSymbol.hpp, Support.cpp to a new folder test_folder
download antlr-2.7.5.exe to test_folder
comment out all lines in CPP_parser.g dealing with antlrTrace()
run antlr-2.7.5.exe CPP_parser.g
run msys
extract antlr-2.7.6.tar.gz
run ./configure --disable-examples && make && make install
change back to test_folder

create test.cpp:

---
#include <iostream>
#include <string>
#include "CPPLexer.hpp"
#include "CPPParser.hpp"

ANTLR_USING_NAMESPACE(std)
ANTLR_USING_NAMESPACE(antlr)

// The following data used by process_line_directive(char*,int) below
// I believe this data and function have to be at this level so as to
// be available to both CPPLexer and CPPParser (and Support.cpp)
int this_line = 0; // current line
 
int deferredLineCount = 0;

int include_line = 0; // include file's line number
int include_last_set = 0; // where included file's line number was last set
char currentIncludedFile[128]; // path and name of current included file

int principal_line = 0; // principal file's line number
int principal_last_set = 0; // where principal file's line number was last set
char principal_file[128]; // path and name of principal file
bool in_user_file = false; // true if we are inside the users's source file

int main()
{
try
{
CPPLexer lexer(cin);
CPPParser parser(lexer);
        parser.init();
parser.translation_unit();
}
catch (exception& e)
{
cerr << "exception: " << e.what() << endl;
}
}

// Needed for #line_number directives generated by gcc -E preprocessing or msvc preprocessing
void process_line_directive(const char *includedFile, const char *includedLineNo)
{
// See global interface variables above
// Working variables
static int line, result;
static bool principal_file_set = false;
static int x;
//printf("Main entered\n");
// Extract included file line no.
result = sscanf(includedLineNo, "%d \n", &line);

//printf("Main: line %d\n",line);
// remove first " from file path+name by shifting all characters left
for(x=1;includedFile[x]!='"';x++)
{
currentIncludedFile[x-1] = includedFile[x];
}

// Check path and name are not too long
if(x>128)
{
//
printf("Path and name of included file too long\n");
printf("Increase length of currentIncludedFile and\n");
printf("  principal_file to at least %d characters\n",x);
printf("Abandon run\n");
getchar();    
}

// Replace last " from file name with null
currentIncludedFile[x-1] = NULL;

if (!principal_file_set)
{
strcpy (principal_file, currentIncludedFile);
principal_file_set = true;
}

// check for main file name
if (strcmp(principal_file, currentIncludedFile) == 0)
{
principal_line = line;
principal_last_set = this_line;
strcpy(currentIncludedFile, " "); // delete include file name
in_user_file = true; // we are processing users's .C or .CPP file (aka principal_file)
}
else
// Check that this is a genuine path
if(currentIncludedFile[1]==':')
{//printf("main.cpp 222 entered\n");
//printf("main.cpp 223 %s %s\n",principal_file,currentIncludedFile);
include_line = line;
include_last_set = this_line;
in_user_file = false; // we are processing a header file
}
}
---

Then compile:
gcc -I/usr/local/include -L/usr/local/lib *.cpp -lstdc++ -lantlr

To test:
run ./a.exe < Quadratic.i (msvc preprocessor processed file)

(gcc -E preprocesses in gcc although not fully supported yet - need to add gcc __attribute__ specifiers and __builtin_va_list to the basic types in the CPP_parser.g grammar)

So it looks fairly complete as a c++ parser.  What would be best is if the grammar is modified to fully support gcc -E for handling preprocessing.  This way there's no bugs as well - tried and true tools like the gcc preprocessor are used to handle #includes, etc.  So code completion would be robust since it would have all the correct symbols defined in the current source file.  And then you could just do gcc -E *.cpp for all cpp project files if global code completion is enabled.
« Last Edit: December 26, 2005, 09:29:04 pm by grv575 »

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: New parser model for Code completion
« Reply #74 on: December 26, 2005, 09:30:58 pm »
I would recommend to use a library for pre-processing:

C/C++ pre-processor parsers:
ucpp
Wave