event handling of our CC, any new idea of the task queue design?

[ollydbg]

We have many timers in our CC code. For example, editor activated event.

The first thing we know is that we don't directly response to those timers, because they happens too often. For example, if user open several editors in the same time, we may receive many such events, but what we are interest in is the last event. To implement this, we have a delay timer for the editor activated event.

We have many other timers, such as editor content changed, mouse position moved, editor closed, project loaded... And you can see that we have a lot of timers, which do quite similar things, they just start a delayed timer, if a similar event happens again(such as a new editor is activated again), then we just restart the timer gain, and all the actual operation is done in the timer event handler. This is the way much similar like the mouse dwell event, we only response to stable events, and thus have better performance.

Events are related, I mean if we first get a mouse position changed event, and when its delayed timer is running, we may received the new editor opened events, at this time, we should cancel the job associated with the mouse position changed event, since we have a new editor opened, and the mouse position is invalid for the new editor. In this situation, we should just sweep the job for mouse position changed event. There are a lot of similar cases of such event relationship.

Idea: I think we need a unified way to implement a TaskQueue, it may have such features:

• when an event happens, we register a new job, the job may run immediately, or we should run the job in a worker thread pool
• when an event happens, we register a new job, but a timer is started
• always keep the last same kind of event, such as editor active event, this means a new same event will cancel a same job
• event may have relations, so an important event may sweep all the previous jobs
• when the timer reached, the job should be run
• when there is not jobs in the queue, the timer may stopped

Any ideas?
Are there some code we can directly use?

EDIT: I think not the native CC, but other CC such as CC for Fortran, Clang CC may benefit.

[yvesdm3000]

I've actually redone and removed a couple of the timers that were originally from Alpha, mostly because there is now a JobQueue that performs some lengtly operations and when that operation is done, the result is, when a lot of UI events come in, often no longer desired and is therefore dropped. You must however keep in mind that most of the Clang-operations are not read-only (sadly not even the code-completion operation since it does a simplified reparse) so they cannot be parallelized. You can parallelize on different translation units, but generally we're only interested in 1 translation unit: The one we're editing. Experiments using 2 translation units for the same document showed that Clang doesn't like it, and we can't copy a translation unit. So we're stuck with 1 JobQueue (except for indexing, could be totally separated because there is no interaction with the UI needed so ThreadPool comes to mind)
I currently designed it so to keep lock contention to a minimum by sending a message to the one job queue and sending a message back for event handling but also for destruction purposes.

Patches and/or suggestions for improvements on ClangCC are always welcome. :-)

Yves

[ollydbg]

I've actually redone and removed a couple of the timers that were originally from Alpha, mostly because there is now a JobQueue that performs some lengtly operations and when that operation is done, the result is, when a lot of UI events come in, often no longer desired and is therefore dropped.

Hi, yvesdm3000, thanks for the reply. You said that old events may be dropped when they are not desired, that's the same thing I said, great! I will look at your git code repo to see how JobQueue works in your clangCC.

You must however keep in mind that most of the Clang-operations are not read-only (sadly not even the code-completion operation since it does a simplified reparse) so they cannot be parallelized.

What do you mean by "not read-only", you mean you can not call two functions inside the clang shared library at the same time from two worker thread?

You can parallelize on different translation units, but generally we're only interested in 1 translation unit: The one we're editing. Experiments using 2 translation units for the same document showed that Clang doesn't like it, and we can't copy a translation unit. So we're stuck with 1 JobQueue (except for indexing, could be totally separated because there is no interaction with the UI needed so ThreadPool comes to mind)

Do you means you have one JobQueue for one translation unit?

Under the native CC, we have one ThreadPool running a lot of parser threads to index all the files inside a cbp project.

I currently designed it so to keep lock contention to a minimum by sending a message to the one job queue and sending a message back for event handling but also for destruction purposes.

Does the JobQueue works on a worker thread?

Patches and/or suggestions for improvements on ClangCC are always welcome. :-)

OK, if I can build the ClangCC plugin, and test it.
I haven't used clang code completion for several years, I see the last time I use clangCC was around 2011, at that time, I use the git repo https://github.com/Lalaland/ClangComplete.git, but I see this project is halt around 2012.

[yvesdm3000]

Hi, yvesdm3000, thanks for the reply. You said that old events may be dropped when they are not desired, that's the same thing I said, great! I will look at your git code repo to see how JobQueue works in your clangCC.

An example is probably better here:
- Document 1 is activated
- translation unit creation job started
- Document 2 is activated
- translation unit creation job finished
 ==> the translation unit created is no longer current for the activated document

What do you mean by "not read-only", you mean you can not call two functions inside the clang shared library at the same time from two worker thread?

When things are read-only (and const), you could read from it from multiple threads. This is not at all applicable here as I explained.

Do you means you have one JobQueue for one translation unit?

No reason to have a JobQueue for one translation unit. It would only help when you switch from one document to another and I don't consider that a problem at all currently.

Under the native CC, we have one ThreadPool running a lot of parser threads to index all the files inside a cbp project.

I'll probably have the same once the indexing code of ALL files is in place, but I start with 1 thread to keep development simple and move to a ThreadPool later.

Does the JobQueue works on a worker thread?
[/quote/
Yes. It's the C::B implementation.

Patches and/or suggestions for improvements on ClangCC are always welcome. :-)

OK, if I can build the ClangCC plugin, and test it.
I haven't used clang code completion for several years, I see the last time I use clangCC was around 2011, at that time, I use the git repo https://github.com/Lalaland/ClangComplete.git, but I see this project is halt around 2012.

I suggest using the one from the 'staging' branch, it's nearly ready to be merged into master, only a double diagnostics-event needs to be fixed in it, which is not too shaby.

Also don't enable the ClangLib toolbar on windows, it makes C::B unstable. I need to look into that shortly (don't understand why, it's an exact copy of the builtin-cc).

https://github.com/yvesdm3000/ClangLib

Yves

[ollydbg]

@Yves, thanks. (I see you forgot adding a quote tag in your reply so that you words are quoted)

I just looked at clanglib's code, I see that
1, when the sdk(ccmanager) ask the suggestion list, you just send the request(a synchronous job) to a worker thread, and wait for the job get done in a limited time
2, I see you use the background thread class, which is a single thread queue
3, you have many components, such as:

Code

    ClangCodeCompletion m_CodeCompletion;
    ClangDiagnostics m_Diagnostics;
    ClangToolbar m_Toolbar;

so that when you get an c::b event, you can search on the component list, and see which one can handle such event.

But I haven't see some code that you can drop some already queued job from the thread queue. So, maybe, I need to implement one...

[yvesdm3000]

But I haven't see some code that you can drop some already queued job from the thread queue. So, maybe, I need to implement one...

Correct. I planned to do this, that's why the JobType enum is really there, but I focussed on optimizing the sequence of events first, and I want the tokendatabase to span the whole project/solution with an indexer to fill it, that way a couple of 'goto' operations could be implemented. There is some ugly code in the 'devel' branch that needs to be cleaned up to do this.

There is also still something that doesn't work on windows, which is overriding the 'tab'-key to perform the task to jump to a function parameter prototype after you did a code-completion. The implementation is pretty hackish but seems to work perfectly on Linux (I'm using this daily), but is very unreliable on Windows. I haven't found a 'clean' way to implement this without altering C::B.

Yves