CC enhancements/alternatives

[MortenMacFly]

Hopefully we don't ever declare a local variable somewhere after a CC_LOCK then.

So what? The worst that can happen is a compiler error in that case... or am I missing something?!

And lastly, hopefully you'll get rid of 90% of these locks (which is the entire point).

Well that's the point, my intention on doing this was to understand why and where all these locks are needed. And in fact, I am a bit frustrated now. So, we need to protect the parser, the tokens tree and the class browser thread from cross-use. therefore three mutexes are floating around. However, accessing the tokens tree and its content is really scattered (for various reason) and I don't see a good way how this could be done better. There are a lot of cases where you need to access the tree and/or the parser in different ways from different sources or you loose CC capabilities.

How on earth could this be done better? We can start a discussion...

BTW: In theory, the construct should be robust enough that we could try 2 or more parsing threads now IMHO.

[oBFusCATed]

However, accessing the tokens tree and its content is really scattered (for various reason) and I don't see a good way how this could be done better.

Have you thought of defining simple public interface?
An interface which have all the features needed by the outside world and not exposing the guts for the parser.

BTW: Is there any actual reason to continue working on our parser? Have you tried the clangs plugin? (I've not, but I think this is the only reasonable direction we should take, when talking about the CC)

[MortenMacFly]

Have you thought of defining simple public interface?

Hehe - sure. That was also the original intention, but you have just way too many interfaces you need. That's the point here.

Is there any actual reason to continue working on our parser? Have you tried the clangs plugin? (I've not, but I think this is the only reasonable direction we should take, when talking about the CC)

I think it's still quite good and fast. The only alternative I see is to make use of Eran's CodeLite and wrap it in a plugin. Instead of wrapping Clang itself this would have been a way better choice (not that I don't like the clang plugin though). However, I don't have time for this and I know it not well enough to honest, so I cannot measure the effort it needs. I already asked in the forums for people willing to try, but nobody responded so far. :-(

Maybe if Eran could give a starter this would be of help (he is interested in it, too)...?!

[oBFusCATed]

I have some little interest, but I fear that I'll get trapped in a major refactoring similar to the one related to the debugger, so I restrain myself from thinking too much about CC

[ollydbg]

I think it's still quite good and fast. The only alternative I see is to make use of Eran's CodeLite and wrap it in a plugin.

CodeLite now integrates the Clang codecompletion feature.
You want to wrap Codelite's CC, do you mean the "original CC in Codelite" or "the total thing" (include Clang cc in Codelite) ?

[MortenMacFly]

CodeLite now integrates the Clang codecompletion feature.

Yes, I know. It seems you can use either one, or even both.

You want to wrap Codelite's CC, do you mean the "original CC in Codelite" or "the total thing" (include Clang cc in Codelite) ?

That would be my first question to Eran: If the interface allows to use both, sure. If it would mean we have to have two plugins then I would start the most promising one first.

However, literally years ago I already started working on that topic and had a C::B plugin ready that at least launched the scanning task using Eran's library. But when integrating with the UI I found myself lost because it required components that were bound to LiteEditor, which we don't have. At that time I stopped. Now I don't know well if the CC part is really de-coupled from the rest of Eran's IDE. That would be the first thing we need to understand (with Eran's help, probably). What are the components we need use, where are the public interfaces not bound into LiteEditor. At the time I tried I used CodeLite, codelite_indexer, CxxParser, sqlite3 and wxsqlite3. I don't know if that has changed though...

Maybe Eran can give an insight...

[oBFusCATed]

As far as I know, the llvm codebase is tailored to make integration easier.
So I don't see any reason to use another layer of abstraction/obfuscation.
I've watched a video about clangs' API and it seemed pretty easy to use.

[eranif]

That would be the first thing we need to understand (with Eran's help, probably). What are the components we need use, where are the public interfaces not bound into LiteEditor. At the time I tried I used CodeLite, codelite_indexer, CxxParser, sqlite3 and wxsqlite3. I don't know if that has changed though...

NOTE: all the below is for my in-house code completion and not for clang

You can build a wrapper around the code completion without the need of the LiteEditor (the exe project).

I actually implemented that when creating my CC unit testing.

The unit test is a simple executable which links with the CC library "libcodeliteu.dll / libcodelite.so"
However, the unit testing are assuming that a sqlite database with all the symbols is already exists for it to use.

You can browse the SVN here:
http://codelite.svn.sourceforge.net/viewvc/codelite/trunk/CodeCompletionsTests/CCTest/cctest.cpp?revision=5407&view=markup

Initializing the library is done like this:

Code

// Here is a simple call flows to initialize the CC of codelite
////////////////////////////////////////////////////////////////

// First, we initialize the TagsManager class
// which is responsible for communicating with the parser thread
// and provides an easy API for the application level

// Set the path to the codelite_indexer binary
// For example, codelite installs codelite_indexer under /usr/bin
// so we path /usr/bin here
TagsManagerST::Get()->SetCodeLiteIndexerPath(wxT("/usr/bin"));

// Start the indexer
// If it will termiante or killed - the TagsManager class will restart it automatically for you
TagsManagerST::Get()->StartCodeLiteIndexer();

// Initialize some settings
wxFileName fn(wxT("/path/to/my/symbols/db.tags"));
TagsManagerST::Get()->OpenDatabase( fn );

// Initialize the parsing thread
// By setting some search paths and event handler to receive the various 
// events

// You can set many flags here have a look at tags_options_data.h to see a complete list of flags
// Next, set some incldue and excelude paths
wxArrayString inclPath;
wxArrayString exclPath;

inclPath.Add(wxT("/usr/include/c++/4.4"));
inclPath.Add(wxT("/usr/include/c++/4.4/x86_64-linux-gnu"));
inclPath.Add(wxT("/usr/include/c++/4.4/backward"));
inclPath.Add(wxT("/usr/local/include"));
inclPath.Add(wxT("/usr/lib/gcc/x86_64-linux-gnu/4.4.3/include"));
inclPath.Add(wxT("/usr/lib/gcc/x86_64-linux-gnu/4.4.3/include-fixed"));
inclPath.Add(wxT("/usr/include"));
inclPath.Add(wxT("/usr/include/qt4/QtCore"));
inclPath.Add(wxT("/usr/include/qt4/QtGui"));
inclPath.Add(wxT("/usr/include/qt4/QtXml"));
inclPath.Add(wxT("/home/eran/wx29/include/wx-2.9"));

// Set the search paths and start the parser thread
ParseThreadST::Get()->SetSearchPaths(inclPath, exclPath);

//=======================================================
// Set the window that will receive the following events:
//=======================================================
// wxEVT_PARSE_THREAD_MESSAGE              - update message from the parsing thread
// wxEVT_PARSE_THREAD_CLEAR_TAGS_CACHE     - in case the application is managing a cache of the symbols, 
//                                           this events instructs the application that now is a good time to clear it
// wxEVT_PARSE_THREAD_SCAN_INCLUDES_DONE   - this event indicates that scanning for include files to parse is completed, a complete list of headers
//                                           to parse are included in the event data
// wxEVT_PARSE_THREAD_RETAGGING_PROGRESS   - Retagging is in progress
// wxEVT_PARSE_THREAD_RETAGGING_COMPLETED  - Retagging is done
// wxEVT_PARSE_THREAD_UPDATED_FILE_SYMBOLS - If you are managing "Tree Symbol" view - this event pretty much tells you to refresh it

// Of course, you set the notifiedWindow to NULL
ParseThreadST::Get()->SetNotifyWindow(NULL);

// Start the parsing thread
ParseThreadST::Get()->Start();

// To communicate with the parser, thread usually you simply call the TagsManager class


// Usage example:
///////////////////////////////////////


// Parsing list of files - quick retag: this means that only modified files since the last
// re tag

std::vector<wxFileName> files; // files are expected to have full path 
                               // this method will not scan all non c-files (or files that their extension does not fit the TagsOptionsData class)

TagsManagerST::Get()->RetagFiles(files, false);

// Full retag: drop all the information stored in the database, and reparse the files
TagsManagerST::Get()->RetagFiles(files, true); 

// Suggest completion for 'wxTheClipboard->'

// The output of the function is stored in 'tags'
std::vector<TagEntryPtr> tags;
TagsManagerST::Get()->AutoCompleteCandidates(wxFileName(wxT("/path/to/source/file/name.cpp")),   // the full path for the input source file
                                             1,                                                  // the line where the caret at
                                             wxT("wxTheClipboard->"),                            // The requested expression to parse
                                             fileContent,                                        // The content of the file, up to do the caret position (in our example, an empty string will do)
                                                                                                 // this is needed to parse for local variables
                                             tags);                                              // The output.

To create an executable, you need to link against:
Debug:

Code

libCodeLiteud.dll;libwxsqlite3ud.dll

Release

Code

libCodeLiteu.dll;libwxsqlite3u.dll

Under linux:

Code

libcodeliteu.so;libwxsqlite3u.so

You can use the codelite_indexer.exe for Windows directly from SVN or build one by pulling codelite's sources and open the workspace /path/to/codelite/sdk/codelite_indexer/codelite_indexer.workspace

The clang completion is a different story and it is tightly coupled with the IDE itself - but this is not an issue for you guys, since you already got one in progress

Eran

[MortenMacFly]

I actually implemented that when creating my CC unit testing.

Thank you, Eran that's interesting to see.

The unit test is a simple executable which links with the CC library "libcodeliteu.dll / libcodelite.so"
However, the unit testing are assuming that a sqlite database with all the symbols is already exists for it to use.

Do you have a similar code snippet for creating the database or point to the direction where this is done to see what's needed?

[eranif]

In the link to the source file I posted above, look at:

Code

void testRetagWorkspace()

In general the steps are:
- Initialize the TagsManager
- Start the codelite_indexer
- Call TagsManagerST::Get()->RetagFiles(...);

Eran

[MortenMacFly]

- Initialize the TagsManager
- Start the codelite_indexer
- Call TagsManagerST::Get()->RetagFiles(...);

OK, thank you.

So, people willing to take action:
Remember that I have a stub for a code completion plugin as C::B plugin project available for anyone who wants to try. Please step forward... ;-)