Ceniza's CodeCompletion Project

[Ceniza]

I've been thinking that maybe, just maybe, I should publish the code of my preprocessor. Questions are: where should I upload it (CB's svn respository, code.google.com, just a zip file somewhere, ...), and, most importantly, is it really going to be used?

I doubt it's going to be used right away in its current state. Chances are some modifications will be needed and some extra information saved during the process. Perhaps some more optimizations may be required, but it would be, at least, a starting point.

What's a real shame is that the document that explains its development is written only in Spanish. Well, you can always learn a new language, so it shouldn't be a problem either :wink:

[JGM]

What's a real shame is that the document that explains its development is written only in Spanish. Well, you can always learn a new language, so it shouldn't be a problem either :wink:

Well people that doesnt know spanish use google translate

[joubertdj]

I've been thinking that maybe, just maybe, I should publish the code of my preprocessor. Questions are: where should I upload it (CB's svn respository, code.google.com, just a zip file somewhere, ...), and, most importantly, is it really going to be used?

I doubt it's going to be used right away in its current state. Chances are some modifications will be needed and some extra information saved during the process. Perhaps some more optimizations may be required, but it would be, at least, a starting point.

What's a real shame is that the document that explains its development is written only in Spanish. Well, you can always learn a new language, so it shouldn't be a problem either :wink:

We need to start somewhere ... this is basically the only thing I currently want in C::B ... if you could post the code here, maybe we could all have a look? Hell if it does Allegro, correctly then I am satisfied it will work with everything!

[ollydbg]

I'm also interested in improving the Code Completion. Especially, pre-process is necessary.
For example:
Now, these code can't be recognized as a function declaration.

Code

CVAPI(void*)  cvAlloc( size_t size );

 
because CVAPI is a macro defined by

Code

  #ifndef CVAPI
    #define CVAPI(rettype) CV_EXTERN_C CV_EXPORTS rettype CV_CDECL
  #endif

Also, there are macros nested.

So, I think pre-process is complicated.
But why not we borrow code from a general c compiler source code?

[Ceniza]

With the information you provided, my preprocessor outputs:

CV_EXTERN_C CV_EXPORTS void * CV_CDECL cvAlloc ( size_t size ) ;

I'll have to check the source code first, and try to isolate a few things. Maybe for this weekend I'll be doing that. I also need to find where I put the test for the optimized string type, and put that code along with it. The doxygen generated documentation may be worth uploading too, and the Spanish document for those who can read it.

Let's hope I don't forget...

[ollydbg]

Great work!
Thank you! I'm looking forward to see the new update.

[Ceniza]

I have started to upload files already. Right now I have only uploaded all documentation files (or the like) I found. I hope to have the source code ready this afternoon (UTC+1), which will be uploaded to the same place.

URL: http://sites.google.com/site/ceniza666/ccpp

[edit]
It looks like it was organized enough to upload it like that
[/edit]

[ollydbg]

I have started to upload files already. Right now I have only uploaded all documentation files (or the like) I found. I hope to have the source code ready this afternoon (UTC+1), which will be uploaded to the same place.

URL: http://sites.google.com/site/ceniza666/ccpp

Great! I'm happily reading the CCP document. It's a new string type. Hopefully I can do some help to improve the plugin.

[ollydbg]

I download the "CCPP.zip" from your site, then open the "CCP.cbp" with codeblocks svn5456 in windows xp.
I can successfully build the project named "Test + Lib -debug" shown  in "build target" toolbar.

When I want to run the "CCPP\bin\Debug\test.exe", by default it's argument is "../../tests/pptest05.cpp"
It complained that it can't find "iostream". So, I manually add the below code to main.cpp.

Code

fsp.addLocalSearchPath(defaultStringType("D:/MinGW/lib/gcc/mingw32/4.3.2/include/c++"));

But it still complained

D:/MinGW/lib/gcc/mingw32/4.3.2/include/c++/iostream:44: 'bits/c++config.h' not f
ound
Process returned 1 (0x1)   execution time : 0.015 s
Press any key to continue.

So, I add another search path

Code

fsp.addLocalSearchPath(defaultStringType("D:/MinGW/lib/gcc/mingw32/4.3.2/include/c++/bits"));

But it still can't open the "c++config.h".

There is the code in iostream , showing that it use a relative path to include "c++config.h" .

Code

...
#ifndef _GLIBCXX_IOSTREAM
#define _GLIBCXX_IOSTREAM 1

#pragma GCC system_header

#include <bits/c++config.h>
#include <ostream>
#include <istream>
...

So, I suspect that your code has bugs to search a relative path. Thanks.

[ollydbg]

Addition.
I modify the pptest05.cpp to check it can preprocess correctly.

Code

#include<cv.h>
#include<cxcore.h>
#include<highgui.h>
using namespace std;
int main() {
    double a[5][5]={
        0,4,6,9,0,
        -5,2,0,1,2,
        -2,0,1,6,7,
        -6,1,1,1,1,
        -1,2,3,0,4
    };
    CvMat ma=cvMat(5,5,CV_64FC1,a);
    CvMat* m=cvCreateMat(5,5,CV_64FC1);
    m=cvCloneMat(&ma);
    CvMat *mb=cvCreateMat(5,5,CV_8UC1);
    cvCmpS(m,0,mb,CV_CMP_LT );
    return 0;
}

and add those search path to main.cpp

Code

    fsp.addLocalSearchPath(defaultStringType("D:/MinGW/lib/gcc/mingw32/4.3.2/include/c++"));
    fsp.addLocalSearchPath(defaultStringType("D:/MinGW/lib/gcc/mingw32/4.3.2/include/c++/bits"));
    fsp.addLocalSearchPath(defaultStringType("F:/opencv/include/opencv"));
    fsp.addLocalSearchPath(defaultStringType("D:/MinGW/include"));
    fsp.addLocalSearchPath(defaultStringType("D:/MinGW/lib/gcc/mingw32/4.3.2/include"));

Then, it reported

src/test/../../tests/pptest05.cpp:19: #endif expected

Process returned 1 (0x1)   execution time : 0.359 s
Press any key to continue.

But I don't think I'm missing #endif in the last line of my test code.

[Ceniza]

After setting all include dirs and defines from the GCC version I'm using in Linux, pptest05.cpp was preprocessed successfully. However, your test file gives me another problem here: /usr/include/bits/endian.h:4: #error "Never use <bits/endian.h> directly; include <endian.h> instead."

Too bad I don't have enough time to check into it

[ollydbg]

Some suggestions:

Why not using " GCC -E" command to generate a "preprocessed file". After that, we can use our parser to read the output file.

See the example here:
http://www.network-theory.co.uk/docs/gccintro/gccintro_36.html

In my environment, I can use this command
 gcc -E test.c -o test.i

Then we could take the parser on "test.i".

The "test.i" has very strict syntax like below:

# linenum filename flags

see the document here:

http://developer.apple.com/documentation/DeveloperTools/gcc-4.0.1/cpp/Preprocessor-Output.html

[ollydbg]

include <endian.h> instead

But I don't want to change the standard header file supplied by mingw. endian.h was include in <iostream> .

Your code contains too many templates. It's a bit hard for me to read .

[Jenna]

I just split this topic, because we went away from Ultimate++.
I did not find a better name, so you have to live with it, or suggest a better one  :roll: .

[Ceniza]

include <endian.h> instead

But I don't want to change the standard header file supplied by mingw. endian.h was include in <iostream> .

Your code contains too many templates. It's a bit hard for me to read .

The error message is reported because a name is not defined. It's a shame I did not consider to include some sort of tracking in the code for debugging purposes. That could be a next step. It could report, for example, include files in a "tree" view (something that lets you know what included what and where and the level of depth), when a define is added, when removed, when an #if/#ifdef/#ifndef is checked and satisfied, when it's ignored, and so on. I must admit that I was in a rush to finish that project, and now I lack the time to work on it again. Just to make it more evil and template abusive, the tracker could become another template parameter, with a no-op tracker as default.

I am really aware I abused templates and macros. Also, just like with any other project, I like to improvise and try to come up with different ways to do things. Since the project was originally conceived as a C++ parser, I also tried to make the project a test case. When I realized it would take way too long to make the parser, I reduced the project just to the preprocessor. I fixed many bugs on the way, but it looks like there are still some in there. Too bad the document that gives more insight on the project is in Spanish. Still, I tried to make the Doxygen documentation as complete as possible.

@Jens: Names are my strongest weakness. You can leave the topic name like that. Also, it was a good idea to split it.

[ollydbg]

@Ceniza

I examed your code a little, and found that the "can't read relative path file" problem comes from this piece of code.
In CCPP\include\ccpfsourceprovider.h start from line 115.

Code

template <class StringType, class StringFunctions>
typename CCPFSourceProvider<StringType, StringFunctions>::DataFetcherType CCPFSourceProvider<StringType, StringFunctions>::getSource(const StringType &filename, const StringType &source, bool global, unsigned int start) throw (ccp::FileNotFound<StringType>)
{
    unsigned int found_at = start;

    if (global)
    {
        typename stringVectorType::const_iterator i = m_globalPaths.begin();

        if (start >= m_globalPaths.size())
        {
            i = m_globalPaths.end();
        }
        else
        {
            std::advance(i, start);
        }

        for (; i != m_globalPaths.end(); ++i, ++found_at)
        {
            StringType path(*i);

            if (path.length() > 0 && path[path.length() - 1] != StringFunctions::dirSeparator)
            {
                path += StringFunctions::dirSeparator;
            }

            path += filename;

            if (StringFunctions::fileExists(path))
            {
                return DataFetcherType(IFStreamFileAbstr<StringType>(path), path, true, found_at);
            }
        }

        start = found_at = 0;
    }

You add the file name to the path. then check whether the file is exist.

Here is the solution in CodeCompletion source code parser.cpp line 815

Code

wxString Parser::GetFullFileName(const wxString& src,const wxString& tgt, bool isGlobal)
{
    wxCriticalSectionLocker lock(s_mutexListProtection);
    wxString fullname(_T("")); // Initialize with Empty String
    if (isGlobal)
    {
        fullname = FindFirstFileInIncludeDirs(tgt);
        if (fullname.IsEmpty())
        {
            // not found; check this case:
            //
            // we had entered the previous file like this: #include <gl/gl.h>
            // and it now does this: #include "glext.h"
            // glext.h was correctly not found above but we can now search
            // for gl/glext.h.
            // if we still not find it, it's not there. A compilation error
            // is imminent (well, almost - I guess the compiler knows a little better ;).
            wxString base = wxFileName(src).GetPath(wxPATH_GET_VOLUME | wxPATH_GET_SEPARATOR);
            fullname = FindFirstFileInIncludeDirs(base + tgt);
        }
    }
    ......

[Ceniza]

IIRC, I decided not to use one of the two methods to search for included files. I do not recall if it was global or local, though.

I think I had also added some code to deal with that situation. It was also a bit of a last minute hack. That's the reason that method receives both filename and source. That's what the while loop that follows (into that if) should do.

[ollydbg]

@Ceniza
I briefly read your pdf(English version ). I can get the conclusion of these points.

First, a memory pool is better and faster, allocate a large memory once is better and faster than allocate small chunks by several times.

Second, smart pointer is something which can avoid the reallocate memory.

Now, I'm concerning the parser, I have read the forum, someone are discussion using a general parser like:

YACC or ANTLR

So, I do a search on Google, I found that the CodeLite IDE's parser use a simplified parser generated by Lex and Yacc. I'm unsure if this parser can be used in CodeCompletion.

By the way, the performance is very important. In my computer(windows Xp, intel dualCore), Parsing the whole codeblocks's source code takes 13 seconds. But an older 2.6G computer takes 24seconds. To solve the performance problem, one solution is store the token tree in the harddisk, then load it when a project file opened.

The current code of the parsing is "hand crafted",  I do suggest we can use a more general way.
Thanks.

 

[Ceniza]

Using a general parser instead of creating a handcrafted one again would help simplify things, to some extent. It's important to know quite well the parser to be able to write all the rules while indicating how to resolve all ambiguities. Furthermore, the parser should integrate nicely with C++, and allow for multi-threading. I gave YACC a try once, and it seems to be able to do all those things, provided you use the right flags. Still, writing all the rules, solving all ambiguities the right way, and even twisting the rules, is a task that requires a lot of time and dedication.

Writing all the rules to at least be able to parse correctly all sorts of variable declarations (including templates and function/method pointers), would be a good start. Being able to store all that information is some sort of structure would be a good second step. That minimum test should be enough to get started. It is also worth mentioning that the parser should also include in the rules all those common extensions to the language, like calling convention specification.

As a personal comment, the only thing I hate about C++ is trying to parse it.

[ollydbg]

I just installed CodeLite and test it's parser. But, I found that it's parser doesn't work as well as codeblocks'.  Especially it can't automatically parser the header file, and many bugs. So, it seems a "handcrafted" parser works better then a standard general parser. . Maybe, it will more time and effort to tune a better YACC grammar rule.

What I concern mostly right now is the macros and preprocesser issue. I do expect your preprecosser could be successfully integrate to the current design to solve this issue. Although we would still use a "handcrafted" but fast parser.

[jfouche]

I just installed CodeLite and test it's parser. But, I found that it's parser doesn't work as well as codeblocks'.  Especially it can't automatically parser the header file, and many bugs.

Hi

I'm using codelite because I found that code completion was better than C::B. Can you give me your feedback about bugs you found in codelite code completion ?
FYI, all project files are parsed automatically, and you can add as many directory you want to the auto completion tool (external libs for example). If not in your case, may be a bug, or a misused of codelite.

[nanyu]

I'm using codelite because I found that code completion was better than C::B...

  me too! I use code::blocks. But I think the codelite's "code completion " was better.

[ollydbg]

@jfouche
I can write a very simple test case.

Code

#include <stdio.h>

int abcdefg;

int main(int argc, char **argv)
{
	printf("hello world\n");
	abcd*******************Here
	return 0;
}

code::blocks can give a list, see the screen shot, but Codelite gives nothing.


[attachment deleted by admin]

[jfouche]

Hi Ollydbg

For me everything is fine, just by pressing CTRL-SPACE.
More over, I would like to say that I love c::b, and I'm always reading the forum, looking for interresting news, specialy about code completion.
--
Jérémie

[attachment deleted by admin]

[ollydbg]

@jfouche

You are right, I'm sorry that I'm new to codeLite, so I don't know the short-cut "ctrl + space". (especially, in my windows system, press ctrl + space will do nothing but change the input method ).

I just do more test, I found that Lex&Yacc works really good!

Also, Code::Blocks can really learn from that.