Author Topic: Code completion using LSP and clangd  (Read 311278 times)

Offline Pecan

  • Plugin developer
  • Lives here!
  • ****
  • Posts: 2813
Re: Code completion using LSP and clangd
« Reply #210 on: October 10, 2022, 06:59:15 am »
I did some extra test of how to show the comments.

Here is the log file from CBclangd_client-xxxxx.log:

Code
...

15:07:41.524 >>> readJson() len:240:
{"id":"textDocument/hover","jsonrpc":"2.0","result":{"contents":{"kind":"plaintext","value":"variable m_TcpFile\n\nType: int\nTCP鎺ユ敹鐨勬暟鎹甛n\nint m_TcpFile"},"range":{"end":{"character":13,"line":1},"start":{"character":4,"line":1}}}}


The log file shows the wrong Chinese words.

Code
 int\nTCP鎺ユ敹鐨勬暟鎹甛n\n

The following patch solves this issue:

Code
 clangd_client/src/LSPclient/client.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/clangd_client/src/LSPclient/client.cpp b/clangd_client/src/LSPclient/client.cpp
index c4a7729..76d19ca 100644
--- a/clangd_client/src/LSPclient/client.cpp
+++ b/clangd_client/src/LSPclient/client.cpp
@@ -1020,7 +1020,7 @@ bool ProcessLanguageClient::readJson(json &json)
     m_MutexInputBufGuard.Unlock();
 
     if (stdStrInputbuf.size())
-        writeClientLog(wxString::Format(">>> readJson() len:%d:\n%s", length, stdStrInputbuf.c_str()) );
+        writeClientLog(wxString::Format(">>> readJson() len:%d:\n%s", length, GetwxUTF8Str(stdStrInputbuf.c_str()).wx_str()) );
 
     // remove any invalid utf8 chars
     bool validData = DoValidateUTF8data(stdStrInputbuf);

In clangd_client rev 83, I've completely reworked writing to the logs.

The purpose of the logs (for me) is to see exactly what clangd was sending to the CB client. My prior code was not accomplishing that. It was converting std:strings (from clangd) to wxStrings. I should not have done that.

I changed the log code to record the std:strings as clangd sent them, which is much more accurate in my estimation.

Would you retest the the error you're getting to see if rev 83 also gets the error.
If it does, I'd rather fix it using the conversion methods you and sodev suggested in other forum messages without modifying the logs.

Thanks for testing.

Offline MaxGaspa

  • Multiple posting newcomer
  • *
  • Posts: 37
Re: Code completion using LSP and clangd
« Reply #211 on: October 10, 2022, 11:17:39 am »

About my replay #203....


Yes, I was finally able to replicate the issue and fix it in the new Nightly 221008.

Pecan, I do not want to bore you....

Yes you fixed the issue related to my message #205 but the bugs in #203 is still present (latest nightly and plugin, Win7 and Win10).

I'm observing the plugin in "sleep state" in several way but the one in #203 seems the simpler test case. The most surprising thing is that I need to close CB to force a reparse...

If you need a test project to replicate the issue let me know.

Thank you anyway for your plugin...is usable and good (IMHO)

MAx

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Code completion using LSP and clangd
« Reply #212 on: October 10, 2022, 03:21:56 pm »

In clangd_client rev 83, I've completely reworked writing to the logs.

The purpose of the logs (for me) is to see exactly what clangd was sending to the CB client. My prior code was not accomplishing that. It was converting std:strings (from clangd) to wxStrings. I should not have done that.

I changed the log code to record the std:strings as clangd sent them, which is much more accurate in my estimation.

Would you retest the the error you're getting to see if rev 83 also gets the error.
If it does, I'd rather fix it using the conversion methods you and sodev suggested in other forum messages without modifying the logs.

Thanks for testing.

Hi, I have just tested this rev83. But it looks like the log still has some error Chinese chars. See the log below:

Code
{"id":"textDocument/hover","jsonrpc":"2.0","method":"textDocument/hover","params":{"position":{"character":4,"line":5},"textDocument":{"uri":"file:///F:/code/test_clangd_client_tipwin/main.cpp"}}}

19:59:55.500 >>> readJson() len:220:
{"id":"textDocument/hover","jsonrpc":"2.0","result":{"contents":{"kind":"plaintext","value":"variable m_AAA\n\nType: int\nABCDEFG\n\nint m_AAA"},"range":{"end":{"character":9,"line":5},"start":{"character":4,"line":5}}}}

19:59:58.665 <<< Hover:
file:///F:/code/test_clangd_client_tipwin/main.cpp,line[1], char[4]

19:59:58.666 <<< Content-Length: 196



{"id":"textDocument/hover","jsonrpc":"2.0","method":"textDocument/hover","params":{"position":{"character":4,"line":1},"textDocument":{"uri":"file:///F:/code/test_clangd_client_tipwin/main.cpp"}}}

19:59:58.750 >>> readJson() len:240:
{"id":"textDocument/hover","jsonrpc":"2.0","result":{"contents":{"kind":"plaintext","value":"variable m_TcpFile\n\nType: int\nTCP鎺ユ敹鐨勬暟鎹甛n\nint m_TcpFile"},"range":{"end":{"character":13,"line":1},"start":{"character":4,"line":1}}}}

I see you have a function named:

std::string StdString_Format(const std::string fmt, ...)

I try to use this patch to avoid the function call, like:

Code
-------------------- clangd_client/src/LSPclient/client.cpp --------------------
index faecc9c..e92c789 100644
@@ -1068,7 +1068,7 @@ bool ProcessLanguageClient::readJson(json &json)
     m_MutexInputBufGuard.Unlock();
 
     if (stdStrInputbuf.size())
-        writeClientLog(StdString_Format(">>> readJson() len:%d:\n%s", length, stdStrInputbuf.c_str()) );
+        writeClientLog(StdString_Format(">>> readJson() len:%d:\n", length) + stdStrInputbuf);
 
     // remove any invalid utf8 chars
     bool validData = DoValidateUTF8data(stdStrInputbuf);

But I still got the same error result of Chinese chars.  :(

Anyway, though the log file contains error Chinese chars, but the tip window shows correct Chinese words. (I'm showing the whole contents of the hoverString in the tip window)

Code
         // Example Hover contents: L"instance-method HelloWxWorldFrame::OnAbout\n\nType: void\nParameters:\n- wxCommandEvent & event\n\n// In HelloWxWorldFrame\nprivate: void HelloWxWorldFrame::OnAbout(wxCommandEvent &event)"
         // get string array of hover info separated at /n chars.
         wxString hoverString = contentsValue;

See image shot below in attachment.

If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Pecan

  • Plugin developer
  • Lives here!
  • ****
  • Posts: 2813
Re: Code completion using LSP and clangd
« Reply #213 on: October 10, 2022, 08:09:14 pm »

About my reply #203....


Yes, I was finally able to replicate the issue and fix it in the new Nightly 221008.

Pecan, I do not want to bore you....

Yes you fixed the issue related to my message #205 but the bugs in #203 is still present (latest nightly and plugin, Win7 and Win10).

I'm observing the plugin in "sleep state" in several way but the one in #203 seems the simpler test case. The most surprising thing is that I need to close CB to force a reparse...

If you need a test project to replicate the issue let me know.

Thank you anyway for your plugin...is usable and good (IMHO)

MAx
@MaxGaspa
Do NOT think that you bore me. I'm grateful for your help.
Thanks for reporting that the max completion matches list is fixed.

I have not yet had time to investigate the Pause/Sleep ("sleep state") bug.
I agree that it's a bug. I'll look at it very soon.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Code completion using LSP and clangd
« Reply #214 on: October 11, 2022, 03:41:12 am »

But I still got the same error result of Chinese chars.  :(


I guess in some stage, the original string(in UTF8) is still translated to the unicode string by my local encoding(GB2312, not the UTF8).

EDIT

Code
// ----------------------------------------------------------------------------
void ProcessLanguageClient::writeClientLog(const std::string& logmsg)
// ----------------------------------------------------------------------------
{
    if (not lspClientLogFile.IsOpened()) return;
    std::string logcr = "";
    if (not StdString_EndsWith(logmsg, "\n"))
        logcr = "\n";
    lspClientLogFile.Write("\n" + GetTime_in_HH_MM_SS_MMM() + " " + logmsg + logcr);
    lspClientLogFile.Flush();
}
// ----------------------------------------------------------------------------
void ProcessLanguageClient::writeServerLog(const std::string& logmsg)
// ----------------------------------------------------------------------------
{
    if (not lspServerLogFile.IsOpened()) return;
    lspServerLogFile.Write(logmsg);
    lspServerLogFile.Flush();

It looks like the wxFFile::Write only accept the wxString, but you passed the std:;string, so the wrong encoding is used for conversion.
« Last Edit: October 11, 2022, 03:58:37 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Code completion using LSP and clangd
« Reply #215 on: October 11, 2022, 04:17:36 am »
Hi, Pecan, my guess is correct.  ;)

Code
-------------------- clangd_client/src/LSPclient/client.cpp --------------------
index e92c789..5c56b26 100644
@@ -626,7 +626,7 @@ void ProcessLanguageClient::writeClientLog(const std::string& logmsg)
     std::string logcr = "";
     if (not StdString_EndsWith(logmsg, "\n"))
         logcr = "\n";
-    lspClientLogFile.Write("\n" + GetTime_in_HH_MM_SS_MMM() + " " + logmsg + logcr);
+    lspClientLogFile.Write("\n" + GetTime_in_HH_MM_SS_MMM() + " " + GetwxUTF8Str(logmsg) + logcr);
     lspClientLogFile.Flush();
 }
 // ----------------------------------------------------------------------------
@@ -634,7 +634,7 @@ void ProcessLanguageClient::writeServerLog(const std::string& logmsg)
 // ----------------------------------------------------------------------------
 {
     if (not lspServerLogFile.IsOpened()) return;
-    lspServerLogFile.Write(logmsg);
+    lspServerLogFile.Write(GetwxUTF8Str(logmsg));
     lspServerLogFile.Flush();
 
     //(ph 2022/02/16)



With this patch, the log file is correctly showing the non-ascii characters.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Pecan

  • Plugin developer
  • Lives here!
  • ****
  • Posts: 2813
Re: Code completion using LSP and clangd
« Reply #216 on: October 11, 2022, 06:36:51 am »
Hi, Pecan, my guess is correct.  ;)

Code
-------------------- clangd_client/src/LSPclient/client.cpp --------------------
index e92c789..5c56b26 100644
@@ -626,7 +626,7 @@ void ProcessLanguageClient::writeClientLog(const std::string& logmsg)
     std::string logcr = "";
     if (not StdString_EndsWith(logmsg, "\n"))
         logcr = "\n";
-    lspClientLogFile.Write("\n" + GetTime_in_HH_MM_SS_MMM() + " " + logmsg + logcr);
+    lspClientLogFile.Write("\n" + GetTime_in_HH_MM_SS_MMM() + " " + GetwxUTF8Str(logmsg) + logcr);
     lspClientLogFile.Flush();
 }
 // ----------------------------------------------------------------------------
@@ -634,7 +634,7 @@ void ProcessLanguageClient::writeServerLog(const std::string& logmsg)
 // ----------------------------------------------------------------------------
 {
     if (not lspServerLogFile.IsOpened()) return;
-    lspServerLogFile.Write(logmsg);
+    lspServerLogFile.Write(GetwxUTF8Str(logmsg));
     lspServerLogFile.Flush();
 
     //(ph 2022/02/16)



With this patch, the log file is correctly showing the non-ascii characters.

There is no need for the logs to show only ascii chars.
I hex dumped that section of the log. Those non ascii chars are each legal utf8 3byte hex representations of one char.

At the point we need to use/display them, we do the GetwxUTF8String() to convert them to wxStrings.
That works just fine.

Please do not convert the logs. I need them to look just the same as clangd sent to us. Especially when trying to document possible bugs.

Thanks for the work you do. I'm grateful.
« Last Edit: October 11, 2022, 06:47:15 am by Pecan »

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Code completion using LSP and clangd
« Reply #217 on: October 11, 2022, 11:44:26 am »
There is no need for the logs to show only ascii chars.
I hex dumped that section of the log. Those non ascii chars are each legal utf8 3byte hex representations of one char.

At the point we need to use/display them, we do the GetwxUTF8String() to convert them to wxStrings.
That works just fine.

Please do not convert the logs. I need them to look just the same as clangd sent to us. Especially when trying to document possible bugs.

Thanks for the work you do. I'm grateful.


Hi, Pecan, I think the using of member function Write from wxWidgets: wxFFile Class Reference:

Code
bool 	Write (const wxString &str, const wxMBConv &conv=wxConvAuto())

which is wrong here.

If you look at the source code, you will see:

Code
// ----------------------------------------------------------------------------
void ProcessLanguageClient::writeClientLog(const std::string& logmsg)
// ----------------------------------------------------------------------------
{
    if (not lspClientLogFile.IsOpened()) return;
    std::string logcr = "";
    if (not StdString_EndsWith(logmsg, "\n"))
        logcr = "\n";
    lspClientLogFile.Write("\n" + GetTime_in_HH_MM_SS_MMM() + " " + logmsg + logcr);
    lspClientLogFile.Flush();
}

In this code, you first construct a std::string arg0:

Code
arg0 = "\n" + GetTime_in_HH_MM_SS_MMM() + " " + logmsg + logcr

Then, the lspClientLogFile.Write() function need a wxString, so the arg0 is convert to wxString(arg0), and the bad thing is it will use the "Locale encodings", in my PC, it could be GB2312. The wrong thing is: logmsg is returned from clangd, which is encoded in UTF8, and the implicit conversion from arg0 to wxString is using Locale encodings(GB2312).

I think if you just need a byte stream, then we need another member function:

Code
size_t 	Write (const void *buffer, size_t count)

This just Writes the specified number of bytes from a buffer.

I hope you can understand my explanation.



If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Pecan

  • Plugin developer
  • Lives here!
  • ****
  • Posts: 2813
Re: Code completion using LSP and clangd
« Reply #218 on: October 14, 2022, 09:45:18 pm »

...Partial quote....

Yes you fixed the issue related to my message #205 but the bugs in #203 is still present (latest nightly and plugin, Win7 and Win10).

I'm observing the plugin in "sleep state" in several way but the one in #203 seems the simpler test case. The most surprising thing is that I need to close CB to force a reparse...

If you need a test project to replicate the issue let me know.

Thank you anyway for your plugin...is usable and good (IMHO)

MAx

@ MaxGaspa

Try the "-2" version below to see if it fixes the "sleep state" error.
It seems you've run across a CB bug where the "compiler finished" event was not being issued and clangd_client got stuck waiting for the compiler to end.

I've made a temporary fix (until the next Nightly) that circumvents the missing event.

Be careful though, the circumvention allows clangd to parse files while the "Run" command is active. You might want to right-click on the project name and "Pause parsing(toggle)" before running your program from the menu run command.

After the "Run" finishes, you can "Pause parsing(toggle)" again to continue parsing where it left off.

Try:

https://sourceforge.net/projects/cb-clangd-client/files/Plugin_Install_Package/Windows_x64/ClangdClientForCBNightly_221008_rev12969_win64-2.zip/download

« Last Edit: October 14, 2022, 09:50:07 pm by Pecan »

Offline Pecan

  • Plugin developer
  • Lives here!
  • ****
  • Posts: 2813
Re: Code completion using LSP and clangd
« Reply #219 on: October 14, 2022, 10:27:38 pm »
@ olldbg RE: Reply #218 on: October 11, 2022, 02:44:26

Thanks, it works for me.
Will be in rev 85.
Code
// ----------------------------------------------------------------------------
void ProcessLanguageClient::writeClientLog(const std::string& logmsg)
// ----------------------------------------------------------------------------
{
    if (not lspClientLogFile.IsOpened()) return;
    std::string logcr = "";
    if (not StdString_EndsWith(logmsg, "\n"))
        logcr = "\n";
    std::string out = "\n" + GetTime_in_HH_MM_SS_MMM() + " " + logmsg + logcr;
    lspClientLogFile.Write(out.c_str(), out.size());
    lspClientLogFile.Flush();
}
// ----------------------------------------------------------------------------
void ProcessLanguageClient::writeServerLog(const std::string& logmsg)
// ----------------------------------------------------------------------------
{
    if (not lspServerLogFile.IsOpened()) return;
    lspServerLogFile.Write(logmsg.c_str(), logmsg.size());
    lspServerLogFile.Flush();


Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Code completion using LSP and clangd
« Reply #220 on: October 15, 2022, 05:36:38 am »
@ olldbg RE: Reply #218 on: October 11, 2022, 02:44:26

Thanks, it works for me.
Will be in rev 85.
Code
// ----------------------------------------------------------------------------
void ProcessLanguageClient::writeClientLog(const std::string& logmsg)
// ----------------------------------------------------------------------------
{
    if (not lspClientLogFile.IsOpened()) return;
    std::string logcr = "";
    if (not StdString_EndsWith(logmsg, "\n"))
        logcr = "\n";
    std::string out = "\n" + GetTime_in_HH_MM_SS_MMM() + " " + logmsg + logcr;
    lspClientLogFile.Write(out.c_str(), out.size());
    lspClientLogFile.Flush();
}
// ----------------------------------------------------------------------------
void ProcessLanguageClient::writeServerLog(const std::string& logmsg)
// ----------------------------------------------------------------------------
{
    if (not lspServerLogFile.IsOpened()) return;
    lspServerLogFile.Write(logmsg.c_str(), logmsg.size());
    lspServerLogFile.Flush();


Good work, this should the correct fix.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Code completion using LSP and clangd
« Reply #221 on: October 15, 2022, 05:47:11 am »
Hi, I'm building clangd_client rev84 with the latest C::B rev 12976.

Do I need to change the line to this:

Code
// ----------------------------------------------------------------------------
void ClgdCompletion::OnCompilerStarted(CodeBlocksEvent& event)
// ----------------------------------------------------------------------------
{
    //If this is a idCompileMenuRun only, do not set compiler is running
    // else we'll hang before nightly rev 12975 fix
    //#warning Developer should remove this condition after nightly for CB rev 12975
    //if (ns_CompilerEventId == XRCID("idCompilerMenuRun")) return;
    GetParseManager()->SetCompilerIsRunning(true);
}

I have to comment out the line

Code
    //if (ns_CompilerEventId == XRCID("idCompilerMenuRun")) return;

Am I correct?
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Pecan

  • Plugin developer
  • Lives here!
  • ****
  • Posts: 2813
Re: Code completion using LSP and clangd
« Reply #222 on: October 15, 2022, 06:56:44 am »
Hi, I'm building clangd_client rev84 with the latest C::B rev 12976.

Do I need to change the line to this:

Code
// ----------------------------------------------------------------------------
void ClgdCompletion::OnCompilerStarted(CodeBlocksEvent& event)
// ----------------------------------------------------------------------------
{
    //If this is a idCompileMenuRun only, do not set compiler is running
    // else we'll hang before nightly rev 12975 fix
    //#warning Developer should remove this condition after nightly for CB rev 12975
    //if (ns_CompilerEventId == XRCID("idCompilerMenuRun")) return;
    GetParseManager()->SetCompilerIsRunning(true);
}

I have to comment out the line

Code
    //if (ns_CompilerEventId == XRCID("idCompilerMenuRun")) return;

Am I correct?

You are correct. That fix is not needed with CB rev 12975 and above.

Thanks for testing.

Offline MaxGaspa

  • Multiple posting newcomer
  • *
  • Posts: 37
Re: Code completion using LSP and clangd
« Reply #223 on: October 15, 2022, 09:36:37 am »

I've made a temporary fix (until the next Nightly) that circumvents the missing event.

Be careful though, the circumvention allows clangd to parse files while the "Run" command is active. You might want to right-click on the project name and "Pause parsing(toggle)" before running your program from the menu run command.

After the "Run" finishes, you can "Pause parsing(toggle)" again to continue parsing where it left off.


@pecan

At the moment I'm unable to replicate the bug, so the bug seems fixed.

Is there any inconvenience leaving the parser working while I'm running the application? I mean not considering the CPU usage?

Max

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6034
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Code completion using LSP and clangd
« Reply #224 on: October 15, 2022, 12:56:32 pm »
Hi, Pecan, In the latested r84 and with the change to using the std::string output of the both function writeClientLog() and writeServerLog().

I found an interesting thing that I got two different encoding output of the Chinese text in the comments.
One place is here, in the function: bool ProcessLanguageClient::WriteHdr(const std::string &in)
I see that the output of the content(usually the textDocument/didOpen message's content the client try to send to the LSP server) will be in GB2312 in my PC.
While, the content received from the server(usually the textDocument/hover message return from the LSP server) will be in UTF8 format.

I just exam the function body: bool ProcessLanguageClient::WriteHdr(), I see it still use the some kinds of std::string to wxString, and later some wxString to std::string, some conversion will use the default local encoding.

For example:
Code
limitedLogOut = wxString::Format("<<< Write():\n%s", wxString(in)).Mid(0,512) + "<...DATA SNIPED BY LOG WRITE()...>";

The variable in to wxString will use wrong encoding.

So, what I think is that we should use std::string inside the WriteHdr().

Even I see there is a function call:

Code
    // Write raw data to clangd server
    #if defined(_WIN32)
        bool ok = m_pServerProcess->WriteRaw( out ); //windows
        if (not ok)
        {
            writeClientLog("Error: WriteHdr() failed WriteRaw()");
            return false;
        }
    #else
        // unix just posts the msg to an output thread queue, so no return code.
        m_pServerProcess->Write( fileUtils.ToStdString(out) );            //unix
    #endif

I think the content sent to the pipe is still in UTF8 encoding. (I just checked the code in src\winprocess\asyncprocess\winprocess_impl.cpp.

Code
bool WinProcessImpl::Write(const std::string& buff) { return WriteRaw(buff + "\r\n"); }

bool WinProcessImpl::WriteRaw(const wxString& buff) { return WriteRaw(FileUtils::ToStdString(buff)); }

bool WinProcessImpl::WriteRaw(const std::string& buff)
{
    // Sanity
    if(!IsRedirect()) {
        return false;
    }
    m_writerThread->Write(buff);
    return true;
}
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.