Using CTags as main parser

[killerbot]

any commits after phase 1 , phase 2 , ... ?
delay from 3 sec to 0.5 sec --> delicious  :lol:

[rickg22]

I'll commit when I get this bug fixed.
Oops, sorry for "hijacking" the thread. Eranif, you may continue now

[Michael]

That's when I get to *START* the rewrite. My current changes (let's call them "phase 1") are optimization ones - got rid of that stupid 3 second delay when reparsing files, minimized the "updating class browser" delay down to 0.5 seconds (on my machine), and the one i'm currently working on is the parser's FindMatches. Unfortunately, in this one, my program segfaults, it seems that the tree structure is corrupted I may need 3 or 4 days to catch the bug

May be you can try to store your search trees into the SQLite db. This could help to prevent corruption. You can check wxSQLite if it would fulfill your needs.

Michael

[rickg22]

That's not necessary now, i got it fixed and commited (rev 1711)

[Michael]

That's not necessary now, i got it fixed and commited (rev 1711)

Good and fast work .

Michael

[Game_Ender]

Any word on the integration of eranif's CodeCompletion implementation?  It looks very exciting, I too had looked a CTags long ago when the discussion first began.  Only eranif did the hard part and actually implemented something with it.  It is also important to note the benifit of having an on disk SQLite database is the fact that there is no "Saving or Loading Cache", in fact there would be no cache at all.  I also believe that SQLite supports Atomic commits so crashing of the application will not corrupt the database, at least in theory.

[eranif]

Hi,

Sorry I didnt have time to work on it yet. later today I will start decoupling the parser code from my old project to make it a standalone component, should not take long maybe a day or two.

Eran

[takeshimiya]

Hi,

Sorry I didnt have time to work on it yet. later today I will start decoupling the parser code from my old project to make it a standalone component, should not take long maybe a day or two.

Eran

[eranif]

Hi,

I started working on decoupling the code parser from my old project, well it seems like alot of work  ..

So, What I have come out so far is:

I created a small app that does the following:

- Create an empty db with all indexes defined
- Parse sample input files (in batch mode) and populate the db

In the main frame there is a function that called: InitDb()
this is the function the one that should intrest you (all the other files, can be ignored for the moment they are used internally or I will use them later when I will update the demo)

In this file what I do is:
- Open the sample input files and loads their data
- Create language parser (e.g. CPP) - this can easily be extended to other languages as well
- Create the ctags interface
- Execute
- Store the results into SQLite

The db class is called
- USDb

The DB is consistent with 4 tables:

- member_table
- function_table
- class_table
- prototype_table (this is special case of prototype function)

to prevent duplicates, there are indexes on the tables - so duplicate entries are not allowed (but of course, same name different prototype is allowed ...)

This example shows how you can use ctags to create a db with all workspace information saved.

Things it can do but I did not had time to present how to do them (but i will):

- It does not show how to get localmember type (qualifier).
- It does not show how to get the current function class name from cursor location (like the two combo boxes in VC71 located on top of the file)
- It does not suggest members/functions list
- It does not show how to parse local scope (ctags cant parse local variables, this is where my internal parser come to help - lex.yy.cpp & USCPPScanner )

Hopefully tomorrow or during the weekend (the weekend starts here tomorrow I will update the sample with more functionality so you will get the impression of the idea.

To run the sample, place the exe "ctags.exe" udner a valid path - it can be found under "SQLite_db/bin/ctags.exe"

In addition, in order to view the results, I am also providing SQLite browser which I found on SF. it is called "SQLite Database Browser" and can be found under "SQLite_db/bin/" or you can use the simple command line "sqlite.exe"

The database name is "./internals/ctags.db" simply open it with the browser and have a look at the results of the parsing.
The parsing input files (it is actually the Tinyxml code) can be found under "Samples/"

The project can be found here:
http://www.eistware.com/wx/ctags/ctags.zip

Let me know if you had problems or questions,
Eran

[Game_Ender]

Awsome keep up the good work.  There is nothing holding back linux integration is there, I assume you just chose to do a windows example for easy of use.

On another note SQLite databases allow access by multiple processes that can read from the database at once, and locks the entire database during writing.  This would make it pretty easy to thread this sort of CodeCompletion.  You would only need to have a thread safe queue of files for the parser worker thread.  The worker thread just sits parsing all the files in the queue, hopefully doing batch writes and not a bunch of small ones to the database.  Then the main thread can query the database whenever it needs needed without fear of any errors occurring.

[eranif]

For keeping the GUI tree up-to-date, I used the following logic:

- The main thread is creating a parser-thread that waits and wakes up every interval
- Once a new workspace is opened, a parsing of all the workspace files is done by the main thread - like in the example
- The editor, (I dont know c::b much, but i guess it uses scintilla) whenever the text is modified (can be caught in SCN_MOFIIED) - puts the text of the current modified file in map which is a pair of <project/fileName> & data, (e.g. let say you have project name prj1 and the file name is main.cpp, the key for the map is prj1/main.cpp and the value in the map is the file data)
- The parser thread wakes up, locks the map - and copy the content of the map to a local map - so the lock is free ASAP
- It then reads from the db all the entries belongs to project + file name 'select * from table where project='prj1' and file='main.cpp')
- It calls ctags to parse the modified file data retrieved from the map
- It preforms 'diff' on the two sets of results: the modified file results returned from ctags vs. the values from the db (sqlite)
- An array of all modified entries is created and sent by the parser thread to the class-view GUI object (wxTreeCtrl) using wxPostEvent() call

In this way the on-going working is not interrupted by the parsing process, the db is always up-to date and consistent.

Btw, the current parsing can handle the following:
Typedefs, enums, namespace, class (including templates), vars, structs, unions, defines and functions

I did it in windows since i work with VC7.1 - I dont think that there is a problem to port it to linux (I am not using MFC, WINAPI or anything similar that can limit the code) - I simply like VC71 so I work with it.

Eran

[afterain]

- It does not show how to get the current function class name from cursor location (like the two combo boxes in VC71 located on top of the file)
I think maybe we can generate a tags file for current file. And we know line of cursor. then we find nearest function's start line. if line of cursor between function's start line and function's end line, we find it.
- It does not suggest members/functions list
I get readtags.c and readtags.h from ctags, readtags is very fast -- "Even for an unsorted 24MB tag file, tag searches take about one second."
It can over tags file and can get all informations(field/class/line etc.) of every entry. If we want to list wxWindow's  members/functions, we can do it by "class" field. If "class" field is same with "wxWindow", then this member/function can add to list.
- It does not show how to parse local scope (ctags cant parse local variables, this is where my internal parser come to help - lex.yy.cpp & USCPPScanner )
Ctags can parse local variables. like this "ctags --C++-kinds=+l --C-kinds=+l -R". You can use "ctags --list-kinds", it will list all tag kinds for all language. field "l" mean local variable.
Example:
test.cpp
int main()
{
   int i;
   wxWindow a;
   a.test(i);
}

void ttse()
{
   bool b;
   map<int, int> t
ctags:
a   .\test.cpp   /^   wxWindow a;$/;"   l
b   .\test.cpp   /^   bool b;$/;"   l
i   .\test.cpp   /^   int i;$/;"   l
main   .\test.cpp   /^int main()$/;"   f
t   .\test.cpp   /^   map<int, int> t;$/;"   l
ttse   .\test.cpp   /^void ttse()$/;"   f