Code completion using LSP and clangd

[Pecan]

I have developed a clangd CB plugin client using Language Server Protocol (LSP).

It's working well.
However, it requires clangd version 12 to work even better.

What is the CodeBlocks policy of distributing an executable, say, clangd.exe v12 with the usual CodeBlock distribution.

CodeLite also distributes clangd with its LSP plugin.

Request for Comments.

[oBFusCATed]

Shipping clangd is a windows thing. I don't think we should do it. Just detect the LLVM installation and use it. Or prompt the user to install it.

[oBFusCATed]

What is the purpose of this LSP plugin? Just provide faster error messages?
In my experience implementing a CC plugin with LSP doesn't work well (don't remember the details).

[ollydbg]

Hi, Pecan, good work!

For me, I think distribute a clangd.dll is OK, or we can distribute a separate plugin package(include the plugin and the clangd.dll). Then the user can just download and unpack it.

If I remember correctly, there is an option in CodeLite IDE, which can let user select which clangd.dll it will use.

Do you have a code repo which others can build/test this plugin?

What is the purpose of this LSP plugin? Just provide faster error messages?
In my experience implementing a CC plugin with LSP doesn't work well (don't remember the details).

Using the LSP, I think at least code completion list will be more precise than the native CC.

[Pecan]

Shipping clangd is a windows thing. I don't think we should do it. Just detect the LLVM installation and use it. Or prompt the user to install it.

Just detecting the installation and using its clangd does not work. I've already tried that. The llvm's versions up to version 8 have no reliable clangd for a CB plugin.

The clangd in llvm versions 8 through 11 are so different from  version 12 that the plugin using them would require two separate plugins or spaghetti code to support them.

CB will have to pick some version between 10 through 12 and program towards that.

I've chosen to work with clangd v12. It can reload the compile commands database without having to kill/restart the LSP server when a user opens a new file.

[Pecan]

What is the purpose of this LSP plugin? Just provide faster error messages?
In my experience implementing a CC plugin with LSP doesn't work well (don't remember the details).

Using  LSP "workspace/publishDiagnostics" with clangd is working well for me. I'ts halved my development time. I no longer have to compile the code to find the errors. They're presented to me as I enter the code or save it.

Clangd supports all the LSP calls plus it's own extensions. So far, I've implemented all CB CC's features via LSP calls except the Symbols browser. Though that's possible to do.

LSP/Clangd is primarily focused on the translation unit (TU). But its still possible to provide a bigger picture by asking for workspace symbols. I haven't tried that (yet).

[Pecan]

...<snip>...
If I remember correctly, there is an option in CodeLite IDE, which can let user select which clangd.dll it will use.

Do you have a code repo which others can build/test this plugin?
...<snip>...

CodeLite allows the user to choose the clang to use, but it prioritizes using clangd v10, which it distributes in dir Runtime/lsp.
Later versions of clangd are very independent of llvm/clang. For example clangd v12 can use many llvm/clang versions.

I don't have a repository yet. Things are still very experimental. But I'll work on that.

[oBFusCATed]

For the record the latest version of clang is 11.1. clang/llvm 12 is some night build. This would fail dramatically on linux if you base your development on this version at least until clang/llvm 12 is actually released in the summer.

Clangd supports all the LSP calls plus it's own extensions. So far, I've implemented all CB CC's features via LSP calls except the Symbols browser. Though that's possible to do.

I know that, CC features required changes to the CCManager API last time I've tried to do it. At least if you want to provide good experience.
I don't think implementing the diagnostics in another info pane tab is a good idea, but this requires another CCManager change which would allow in-editor-notifications.

Also I don't know how is using a mix of clang/gcc these days, but it was a problem in the past.
Also I'm not sure if it is a good idea to base your LSP plugin on only one implementation. If you're going to do it name it clangd-lsp.

[Pecan]

For the record the latest version of clang is 11.1. clang/llvm 12 is some night build. This would fail dramatically on linux if you base your development on this version at least until clang/llvm 12 is actually released in the summer.

I know that, CC features required changes to the CCManager API last time I've tried to do it. At least if you want to provide good experience.
I don't think implementing the diagnostics in another info pane tab is a good idea, but this requires another CCManager change which would allow in-editor-notifications.

Also I don't know how is using a mix of clang/gcc these days, but it was a problem in the past.
Also I'm not sure if it is a good idea to base your LSP plugin on only one implementation. If you're going to do it name it clangd-lsp.

In the furture, I'm sure we can generalize to a LSP UI. For the time being, I'm coding a plugin to concentrate learning and experimentation. I'll be happy to name it clangd-lsp. That's a good name.
 
Good to know clangd v12 rc1 may be out in the summer. I don't expect this plugin will be ready to publish any sooner.

[Lazauya]

Do you have a public repo? I'm super interested in contributing to this plugin.

[Pecan]

Do you have a public repo? I'm super interested in contributing to this plugin.

I do not yet have a public repo.
The current code is very experimental and in a raw combination of both used and deprecated code, full of unreleasable log writes.

I will make a repo when I can clean up the code  and after I do more debugging.
I'm working hard at it.

But thanks for the offer. 

[Lazauya]

The current code is very experimental and in a raw combination of both used and deprecated code, full of unreleasable log writes.

I will make a repo when I can clean up the code  and after I do more debugging.

I don't mind. I could definitely help in debugging, test writing, code cleaning, etc. This is the kind of thing I've been looking for, and I'd very much like to accelerate development of it as much as possible, if you would let me.

[Pecan]

@ AndrewCot:

What is your concurrently processing threads set to at
Settings->clangd_client-> C/C++parser ->Concurrently parsing threads.

It might not be there. I don't know when I implementd that.

Clangd seems to need 3-4 seconds to index a 2-3k line file, but it's very fast at the user response level once a file is indexed

In the new implementation (not uploaded yet) I set clangd parsing the active editor first, then the other open editors, then recently modified files, then all the others.

That way, you do not have to wait to begin using clangd. Just open the editor you want to work in. It gets sent to the front of the line, and it's should be parsed and indexed in 3 or 4 seconds.

Clangd can parse, Goto (decls and defs), find references, rename symbols etc. etc. all at the same time. No need to wait. Just open the file.

You do not need to wait for files to parse like the older CodeCompletion. Just activate the file you want to work in. The non-opened files parse in another address space at their own pace. If you switch editors, clangd-client knows whether it's been parsed or not. If it hasn't been, it's send immediately to clangd which parses it ahead of any background files.

This works great as long as at least one clangd thread is available.

The trick I found was to set the clangd threads to no more than 1/2 of the system (cpu) threads (hard coded in client allocation). I've given clangd 4 threads of my i7's 8 , and set concurrent parsing threads to 2. That usually guarantees that a thread will be available for priority (active editor) parses.  I'm able to compile, background parse, GoTo, Find references, code complete, show function, etc all at the same time.

My parsing times look 1/2 of yours.
What's your hardware look like? What cpu, how many threads?

But all of this will come to light with testing and discussion (as you are doing now). Thanks.

Go easy on me for a little longer. I'm running into problems.
Clangd is passing back 3 byte UTF8 chars during code completions, crashing json parses. I'm stalled trying to fix that. I found the culprit today, Hopefully I can fix it tomorrow .

@pecan
It appears that processing the files is slow as per the logged info shown below:

Opening D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\CodeBlocks_wx31_64.cbp
Done.
cbProject::Open took: 0.725 seconds.
ProjectManager::SetProject took: 1.686 seconds.
ProjectManager::LoadProject took: 2.732 seconds.
ParseManager::CreateParser: Finish creating a new parser for project 'Code::Blocks wx3.1.x (64 bit)'
LSP opened editor parse finished for D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\plugins\clangd_client\src\LSPclient\include\protocol.h (1441 ms) (496 more)
LSP opened editor parse finished for D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\plugins\clangd_client\src\codecompletion\parser\LSP_tokenizer.h (3194 ms) (495 more)
LSP opened editor parse finished for D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\plugins\clangd_client\src\codecompletion\codecompletion.cpp (4342 ms) (494 more)
LSP opened editor parse finished for D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\plugins\clangd_client\src\LSPclient\include\client.cpp (3738 ms) (493 more)
LSP opened editor parse finished for D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\include\compilerfactory.h (660 ms) (492 more)
LSP opened editor parse finished for D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\plugins\clangd_client\src\asyncprocess\asyncprocess.cpp (2228 ms) (491 more)
LSP opened editor parse finished for D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\sdk\configmanager-revision.cpp (3543 ms) (490 more)
LSP opened editor parse finished for D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\sdk\cygwin.cpp (3534 ms) (489 more)
LSP background parsing finished for: D:\Andrew_Development\WorkingOnThese\AC-WindowsInstaller\src\src\environmentsettingsdlg.cpp (3262 ms) (488 more)

Is there a way of speeding this up? Any ideas on where to look or what to check or do?
I am using MSYS2 with the latest updates as of a few hours ago (this resulted in waht looked like a big MinGW 64 9.00 run time update) and GCC 11.2.0.

[Pecan]

Does anyone know of a good UTF8 char validator.
Clangd is handing back (occasionally) a 3 byte utf8 char.
It's blowing nlohmann json parser out of the water.

I'd like to scan the clangd stdout responses for invalid UTF8.
They're weird. \xE2\x80\xA6 right in the middle of an empty function() param area. Also that strange period half way up the middle of a char area. Like an item list indicator. \xE2\x80\xA2.

When, in the initialization, we tell clangd to use "Only UTF8", it's supposed to mean 8 bits only, not 24 bits. And evidently they don't know it, because they set the response length as if all chars are 8bit.

Thanks

[AndrewCot]

@pecan.

I am not trying to be a PITA, but trying to give feedback and what I have seen when I see it so I do not forget. Even as it is now it's still a big improvement on the existing codecompletion code. The changes I have made are in the https://github.com/acotty/codeblocks_sf/tree/AC-WindowsInstaller branch if you need or want to look at what I have done.

My setup is:
    Windows 10 21H2 with latest updates (checked last night and it was up to date)
    32GB DDR4 3600Mhz ram (2x16gb sticks)
    AMD Ryzen 7 5700G (8 core plus 8 HT for a total of 16, base 3800 Mhz boost 4650)
    No separate GPU
    C: & D: are on a Samsung SSD 980 pro.
    MSYS2 MinGW64 GCC 11.2.0
   C: has the MSYS 2 compiler on it.
   D: has the CB and other source code on it.
 
I had 1 concurrent parsing thread setup, but changed this about 2 hours ago to 8 and it speed up the overall time, but the time shown per file was still about the same.

As for the UTF8 I have no idea, but will do some searching after dinner (OZ Sydney time).