Clang command line support for codecompletion

[ollydbg]

Here is my detailed test for codecompletion in clang (Windows)

prerequisite:

You need to download the package supplied by jens: clang_win32
Then add the path of clang.exe in your PATH.

test

create a file like:
E:\clang_win32\main.cpp
the file contains some code snippet like:

Code

class AAA
{
public:
   int aaa;
   int bbb;
};


int myfunction()
{
  AAA a;
  a.
  return 0;
}

Now, you can see, the  codecompletion list should generated after "a." .

You just run the command: pay attention to the :13:5, this this the line and column number of the caret after "a."

Code

clang -w -fsyntax-only -Xclang -code-completion-at=E:\clang_win32\main.cpp:13:5 E:\clang_win32\main.cpp

Then here is the wonder result:

Code

E:\clang_win32>clang -w -fsyntax-only -Xclang -code-completion-at=E:\clang_win32
\main.cpp:13:5 E:\clang_win32\main.cpp
COMPLETION: AAA : AAA::
COMPLETION: aaa : [#int#]aaa
COMPLETION: bbb : [#int#]bbb
COMPLETION: operator= : [#AAA &#]operator=(<#const AAA &#>)
COMPLETION: ~AAA : [#void#]~AAA()

 

then main reference is eranif's codelite source (thanks eranif):
http://codelite.svn.sourceforge.net/viewvc/codelite/trunk/LiteEditor/clang_code_completion.cpp?revision=4207&view=markup

[ollydbg]

about the pch. suppose I have two files: testcpp.h and main.cpp

testcpp.h

Code

class BBB
{
public:
int f1();
int f2();
int f3();
};

main.cpp

Code

#include "testcpp.h"

int myfunction()
{
  BBB a;
  a.
  return 0;
}

Now,
first I need to generate the pch file using command:

Code

E:\clang_win32>clang++ -cc1 -x c++-header testcpp.h -emit-pch -o testcpp.h.pch

Then, do the code completion command.

Code

E:\clang_win32>clang++ -cc1 -w -fsyntax-only -include-pch testcpp.h.pch -code-co
mpletion-at=E:\clang_win32\main.cpp:8:5  E:\clang_win32\main.cpp
In file included from E:\clang_win32\main.cpp:3:
E:\clang_win32\testcpp.h:3:7: error: redefinition of 'BBB'
class BBB
      ^
E:\clang_win32\testcpp.h:3:7: note: previous definition is here
class BBB
      ^
1 error generated.
COMPLETION: BBB : BBB::
COMPLETION: f1 : [#int#]f1()
COMPLETION: f2 : [#int#]f2()
COMPLETION: f3 : [#int#]f3()

Strange that there is an error about "redefinition".

So, I just comment the #include line like, so the modified main.cpp

Code

//#include "testcpp.h"

int myfunction()
{
  BBB a;
  a.
  return 0;
}

Now, everything works fine.

Code

E:\clang_win32>clang++ -cc1 -w -fsyntax-only -include-pch testcpp.h.pch -code-co
mpletion-at=E:\clang_win32\main.cpp:8:5  E:\clang_win32\main.cpp
COMPLETION: BBB : BBB::
COMPLETION: f1 : [#int#]f1()
COMPLETION: f2 : [#int#]f2()
COMPLETION: f3 : [#int#]f3()

Any comments about how to solve the redefinition issue???

thanks.

[ollydbg]

Sorry guys, I have made a mistake in my previous post, now add the include guard solved all the problem of redefinition.

modified testcpp.h

Code

#ifndef TESTCPP_H
#define TESTCPP_H

class BBB
{
public:
int f1();
int f2();
int f3();
};

#endif

@eranif

it seems the clang++ and PCH works fine. You can try it to see if it works much faster.

[Jenna]

@jens:
thanks for your time and effort to build the win32 version of clang for me, it works quite well.

Nice to hear, for me it's much easier (the most time) to do a cross-build for windows in my linux, than do the build on windows itself.

[ollydbg]

I have just briefly read the Clang's source code about
lexer
parser
And I found that it is definitely all hand-written code (Note our CC's parser is also hand-written), but it is quite modular.

It's lexer's source code was located in

Code

llvm\tools\clang\include\clang\Lex
llvm\tools\clang\lib\Lex

parser's source is located in

Code

llvm\tools\clang\include\clang\Parse
llvm\tools\clang\lib\Parse

for example:
In the parser.cpp

Code

/// \brief Determine whether the current token, if it occurs after a
/// declarator, continues a declaration or declaration list.
bool Parser::isDeclarationAfterDeclarator() const {
  return Tok.is(tok::equal) ||      // int X()=  -> not a function def
    Tok.is(tok::comma) ||           // int X(),  -> not a function def
    Tok.is(tok::semi)  ||           // int X();  -> not a function def
    Tok.is(tok::kw_asm) ||          // int X() __asm__ -> not a function def
    Tok.is(tok::kw___attribute) ||  // int X() __attr__ -> not a function def
    (getLang().CPlusPlus &&
     Tok.is(tok::l_paren));         // int X(0) -> not a function def [C++]
}

there are even skip functions in lexer ( we have skip XXX functions in CC either)

Code

/// SkipEscapedNewLines - If P points to an escaped newline (or a series of
/// them), skip over them and return the first non-escaped-newline found,
/// otherwise return P.
const char *Lexer::SkipEscapedNewLines(const char *P) {
  while (1) {
    const char *AfterEscape;
    if (*P == '\\') {
      AfterEscape = P+1;
    } else if (*P == '?') {
      // If not a trigraph for escape, bail out.
      if (P[1] != '?' || P[2] != '/')
        return P;
      AfterEscape = P+3;
    } else {
      return P;
    }

    unsigned NewLineSize = Lexer::getEscapedNewLineSize(AfterEscape);
    if (NewLineSize == 0) return P;
    P = AfterEscape+NewLineSize;
  }
}

BTW: seems all the tokens were handled as char type.(one byte). Hope our CC can use char type instead. because wxChar seems waste a lot of space. (Unicode wxChar seems equal two bytes, not sure how long in X86-64 system).

Another similaity is that Clang do the pre-processors and lexer in the same stage, this is the conceptional same as in CC's Tokenizer.

[ironhead]

BTW: seems all the tokens were handled as char type.(one byte). Hope our CC can use char type instead. because wxChar seems waste a lot of space. (Unicode wxChar seems equal two bytes, not sure how long in X86-64 system).

This is limiting, isn't it?  It would limit CC to be UTF-8 compliant at best (i.e. not truly Unicode).

As far as I know, Unicode is a two-byte sequence irregardless of architecture.

[oBFusCATed]

sizeof(wchar_t) is 4 on MacOSX.
Using non ASCII characters in the sourcecode/sourcefiles is highly discouraged anyway...

[eranif]

BTW: As promised, here are the build instruction for building clang under Windows / MinGW

Important notes:
----------------
MinGW\bin directory must be in the PATH environment variable

Install CMAKE from here:
------------------------
http://www.cmake.org/cmake/resources/software.html

Building clang using MinGW:
--------------------------

- Checkout llvm sources             : svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
- cd llvm/tools directory           : svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
- create a build directory under the root llvm directory (.e.g mkdir build_release)
- cd to the build directory (e.g. cd build_release) and run : cmake -G "MinGW Makefiles" ../ -DCMAKE_BUILD_TYPE=Release
- open the file CMakeCache.txt and reduce the optimizatin level from -O3 to -O2 (replace all -O3 to -O2)
- Open the file llvm\tools\clang\tools\c-index-test\CMakeLists.txt and comment out the following lines:

#add_clang_executable(c-index-test
#  c-index-test.c
#  )

#set_target_properties(c-index-test
#  PROPERTIES
#  LINKER_LANGUAGE CXX)

- Run                               : mingw32-make

To update sources from svn, you need to update both repositories, so:
--------------------------------------------------------------------
from the root directory, run: svn up
cd to tools/clang and run   : svn up

Eran

[ollydbg]

thanks eranif for the information!!!

seems c-index-test has some problems ( I failed building this target the day before yesterday ) see build failed method two..
It seems we don't have perl to build clang. (as the http://llvm.org/docs/GettingStarted.html#requirements said, under MSYS, perl is need), but the cmake+mingw method, perl is not need.


BTW:
As this post said:
http://lists.trolltech.com/pipermail/qt-creator/2009-January/001329.html
In QT creator, they use two kind of parser.
let's say, a simple parsing and a detailed parsing.

When QTcreator analysis the whole project files, it just do the simple parsing. this is the same way as we do in C::B's CC parser. or the Codelite's ctags parser. it just correct tags.

But QTcreator will do a detailed parsing on the current active translation unit. including the type checking, macro expansion...So, the error information can be shown, and a more precise code-completion list can be generated.

As this way: we can just use clang do the "detailed parsing" and left the other thing still to CC's parser or ctags. Any ideas?

[eranif]

Actually, for codelite I took another path:
- All code completion goes through the current parser code
- In case the current parser fails to parse / yield results, the clang code completion "kicks in"

I think that this is the best option in terms of stability and backward compatibilty
I believe in making evolution, not a revolution


Eran
 

[ollydbg]

I have successfully build the clang under msys/mingw with some clang maillist guy's help.



The problem:

updated:

Today, I have tried the method listed in
http://clang.llvm.org/get_started.html

to build a clang.exe, but no luck, all failed   .

1, one failed method.
under Msys mingw, check out the llvm and clang svn code, and configure. make. but the error occurred when I run the make,  such as:

$ make
Makefile:130: /Makefile.rules: No such file or directory
make: *** No rule to make target `/Makefile.rules'.  Stop.

Was solved because I have made a big mistake.

There are two kind of make.exe. One is under mingw/bin, the other is under msys/bin
The error caused because msys use the make.exe under mingw/bin. this was totally wrong, we should use make.exe under msys/bin.
So, you should delete make.exe under mingw/bin or rename it.

The final report:
it takes one hour to compile the total target..( too long..)
it seem the total build file was quite large : 3.5G !!!

and the folder \build\Debug+Asserts is 2.5G.
especially, under the folder: \build\Debug+Asserts\bin

clang++.exe   430M
clang.exe       430M
llc.exe             170M
....

striping the clang++.exe will reduce its size to 21M.

Testing the code-completion feature works fine!!!

[infinigon]

it seem the total build file was quite large : 3.5G !!!

and the folder \build\Debug+Asserts is 2.5G.
especially, under the folder: \build\Debug+Asserts\bin

clang++.exe   430M
clang.exe       430M
llc.exe             170M
....

striping the clang++.exe will reduce its size to 21M.

According to http://llvm.org/docs/MakefileGuide.html#variables you can set ENABLE_OPTIMIZED=1 when make'ing in order to disable debugging symbols and enable optimization (should have no need to strip after this, and clang runs much faster). Also, AFAIK clang.exe and clang++.exe are exactly the same - you only need one copy of it.

Also I tried building clang, but did not realise until too late that if using MSVC to build, then clang would be configured to use the MSVC headers, which it cannot parse. Now I am also trying to build it with MinGW...

[ollydbg]

it seem the total build file was quite large : 3.5G !!!

and the folder \build\Debug+Asserts is 2.5G.
especially, under the folder: \build\Debug+Asserts\bin

clang++.exe   430M
clang.exe       430M
llc.exe             170M
....

striping the clang++.exe will reduce its size to 21M.

According to http://llvm.org/docs/MakefileGuide.html#variables you can set ENABLE_OPTIMIZED=1 when make'ing in order to disable debugging symbols and enable optimization (should have no need to strip after this, and clang runs much faster). Also, AFAIK clang.exe and clang++.exe are exactly the same - you only need one copy of it.

Also I tried building clang, but did not realise until too late that if using MSVC to build, then clang would be configured to use the MSVC headers, which it cannot parse. Now I am also trying to build it with MinGW...

Hi, thanks for your help. I will try to build a ENABLE_OPTIMIZED=1 version if I have free time.
 currently, not sure how "easy/difficult" to add the clang to codecompletion plugin. or making another codecompletion plugin(difficult too).

[ollydbg]

Information for those who would like to use Clang for codecompletion and code indexer.
There are two ways:
1, using the clang from the command line. this is currently done in vim, Emacs, codelite by calling the clang.exe from command line.
2, directly link to the libraries supplied by libclang. this is done only in Apples Xcode, but it is suggest the better way. You can take the file:

Code

\llvm\tools\clang\tools\c-index-test\c-index-test.c

, this file contains all the information you need.

BTW: to build a release version of clang, the options can be found here:
How To Release LLVM To The Public

Code

Building the Release
The build of llvm, llvm-gcc, and clang must be free of errors and warnings in both debug, release+asserts, and release builds. If all builds are clean, then the release passes build qualification.

   1. debug: ENABLE_OPTIMIZED=0
   2. release+asserts: ENABLE_OPTIMIZED=1
   3. release: ENABLE_OPTIMIZED=1 DISABLE_ASSERTIONS=1

Edit:
The common feature like "find the definition of a function" or "find the reference" can be found there:
The Index Library
see below:

index-test tool
Usage
A command-line tool that exercises the libIndex API, useful for testing its features. As input it accepts multiple AST files (representing multiple translation units) and a few options:

   -point-at  [file:line:column]

Resolves a [file:line:column] triplet into a ASTLocation from the first AST file. If no other option is specified, it prints the ASTLocation. It also prints a declaration's associated doxygen comment, if one is available.

   -print-refs

Prints the ASTLocations that reference the declaration that was resolved out of the [file:line:column] triplet

   -print-defs

Prints the ASTLocations that define the resolved declaration

   -print-decls

Prints the ASTLocations that declare the resolved declaration

Examples

Here's an example of using index-test:

We have 3 files,

foo.h:

extern int global_var;

void foo_func(int param1);
void bar_func(void);

t1.c:

#include "foo.h"

void foo_func(int param1) {
  int local_var = global_var;
  for (int for_var = 100; for_var < 500; ++for_var) {
    local_var = param1 + for_var;
  }
  bar_func();
}

t2.c:

#include "foo.h"

int global_var = 10;

void bar_func(void) {
  global_var += 100;
  foo_func(global_var);
}

You first get AST files out of t1.c and t2.c:

$ clang -emit-ast t1.c -o t1.ast
$ clang -emit-ast t2.c -o t2.ast

Find the ASTLocation under this position of t1.c:

[...]
void foo_func(int param1) {
  int local_var = global_var;
                      ^
[...]

$ index-test t1.ast -point-at t1.c:4:23
> [Decl: Var local_var | Stmt: DeclRefExpr global_var] <t1.c:4:19, t1.c:4:19>

Find the declaration:

$ index-test t1.ast -point-at t1.c:4:23 -print-decls
> [Decl: Var global_var] <foo.h:1:12, foo.h:1:12>

Find the references:

$ index-test t1.ast t2.ast -point-at t1.c:4:23 -print-refs
> [Decl: Var local_var | Stmt: DeclRefExpr global_var] <t1.c:4:19, t1.c:4:19>
> [Decl: Function bar_func | Stmt: DeclRefExpr global_var] <t2.c:6:3, t2.c:6:3>
> [Decl: Function bar_func | Stmt: DeclRefExpr global_var] <t2.c:7:12, t2.c:7:12>

Find definitions:

$ index-test t1.ast t2.ast -point-at t1.c:4:23 -print-defs
> [Decl: Var global_var] <t2.c:3:5, t2.c:3:18>

All the features was done in the code:

Code

\llvm\tools\clang\tools\c-index-test\c-index-test.c

This is quite GOOD for an IDE!!!!

[ollydbg]

It also lacks some crucial information (which I could not seem to get an access to, like an exact position of the match), I can probably obtain more information if I will link directly with the clang libraries - but for the POC (prove of concept) it seems too much for me.

Hi, eranif, I have just dig into the libclang, especially the c-index-test.c file to see if is fit your desire.

From the latest svn code of clang and llvm, I found that:

F:\llvm_build\llvm\tools\clang\include\clang-c\Index.h

Code

/**
 * \brief A single result of code completion.
 */
typedef struct {
  /**
   * \brief The kind of entity that this completion refers to.
   *
   * The cursor kind will be a macro, keyword, or a declaration (one of the
   * *Decl cursor kinds), describing the entity that the completion is
   * referring to.
   *
   * \todo In the future, we would like to provide a full cursor, to allow
   * the client to extract additional information from declaration.
   */
  enum CXCursorKind CursorKind;

  /**
   * \brief The code-completion string that describes how to insert this
   * code-completion result into the editing buffer.
   */
  CXCompletionString CompletionString;
} CXCompletionResult;

You can see, the code completion  result (one entry) only contains two members:
one is the CursorKind. and if it supply a "full Cursor" information, you can get the Location information about this code completion entry.
Maybe, the clang developers will improved this in the future.