Allowing Plugin Interdependency and Improving Plugin Management

[dmoore]

Making a new thread based on the discussion started here.

The main ideas:
1. Only load plugin shared library objects for plugins that are enabled
2. Allow plugins to interact with each other. e.g. a compiler add-on plugin might use methods provided by the compiler plugin.

Attached is a patch for #1, which is a work in progress. PluginManager does some stuff with member variables being shared across functions that make it prone to bugs when making changes and, ideally, that should be fixed as part of any overhaul.

I am working on part 2.

[dmoore]

Patch updated to handle safemode

[dmoore]

Some additional thoughts on plugin dependencies

At least on Linux It looks like it is possible to have a plugin compile against another plugin and use its plugin instance without too much trouble if the SOs/DLLs will be in the same place. See the attached plugin that is made to depend on the Compiler plugin (you will need to change some of the hardcoded paths in "Build Options" -> "Search Directories").

Just need to include the relevant plugin headers

Code

#include <compilergcc.h>

and trivial usage:

Code

void Dep_Plugin::OnAttach()
{
    cbPlugin *plug = Manager::Get()->GetPluginManager()->FindPluginByName(_T("Compiler"));
    if (plug)
    {
        cbCompilerPlugin *cplug = dynamic_cast<cbCompilerPlugin*>(plug);
        bool attached = cplug->IsAttached();
        Manager::Get()->GetLogManager()->Log(wxString::Format(_T("Dep plugin found compiler plugin instance %i"),attached));
    }
}

So I think that the only changes to the SDK that are needed:
1. make sure that our dependent plugins are loaded after the plugins they depend on
2. don't load plugins if the plugins they depend on aren't loaded
3. unload plugins if the plugins they depend on are disabled/removed


[attachment deleted by admin]

[oBFusCATed]

At least on Linux It looks like it is possible to have a plugin compile against another plugin and use its plugin instance....

This is because on linux we use -fvisibility=default and this will change in the future, when I have some free time to play with this.
On windows the default is -fvisibility=hidden, so this won't work anyway.

Just need to include the relevant plugin headers

Code

#include <compilergcc.h>

This is bloody dangerous. Nothing in this header is intended for external plugin access.
If you want to access code from other plugins, you'll have to define a clear interface which can be guaranteed to be relatively stable.

I'm not sure also that it is a good idea to allow plugins to depend on other plugins
and I think if something should be accessed by more plugins it is better to provide a SDK API for it.
This way we have a bit clearer interface. But this is just me.

[oBFusCATed]

On a separate topic:
Have you measured the time needed to load C::B with and without your patch.
Currently C::B loads pretty slow if all contrib plugins are installed, but disabled.
Does you patch improve the load time?

[dmoore]

On a separate topic:
Have you measured the time needed to load C::B with and without your patch.
Currently C::B loads pretty slow if all contrib plugins are installed, but disabled.
Does you patch improve the load time?

Some, C::B still has to read all of the manifests, but no DLLs for disabled plugins get loaded. I will do some timings when I have time.

[dmoore]

At least on Linux It looks like it is possible to have a plugin compile against another plugin and use its plugin instance....

This is because on linux we use -fvisibility=default and this will change in the future, when I have some free time to play with this.
On windows the default is -fvisibility=hidden, so this won't work anyway.

That's obviously a choice you can make per plugin. But what are you proposing as the alternative? The way wxSmith does it? i.e. A separate lib for common stuff that all plugins import. That sucks for deployment because what I found was that the common lib needs to be in the same place as libcodeblocks (the way we do it on windows) or in one of the system lib paths (unix). So much for manage plugins -> install new...

Just need to include the relevant plugin headers

Code

#include <compilergcc.h>

This is bloody dangerous. Nothing in this header is intended for external plugin access.
If you want to access code from other plugins, you'll have to define a clear interface which can be guaranteed to be relatively stable.

Sure, my intention was only to figure out what's possible now NOT how it should work. I don't think it would be terribly complicated expose the relevant parts of the compiler plugin to another plugin.

I'm not sure also that it is a good idea to allow plugins to depend on other plugins

Not what you said in the other thread...  

and I think if something should be accessed by more plugins it is better to provide a SDK API for it.

If you mean adding to the C::B SDK, that just bloats C::B unnecessarily e.g Most of the devs couldn't give a shit about python support, but there are a bunch of common things in my python plugins (e.g. XMLRPC stuff) that I would like all of my plugins to be able to use. No reason to have this in the C::B SDK, but I would still like some method of deploying this shared library using our existing plugin management mechanisms

Another example is Tools+. Initially I had the part that provides that notebook tabs built as a separate library so that more than one plugin could define windows that would be displayed there (e.g. output from shell commands, a python interpreter). There is no reason this necessarily needs to be part of the SDK if it isn't being used.

Curious to hear what others think. I won't spend any more time on this if enough of you think this is a dead end.

[oBFusCATed]

Not what you said in the other thread...  

I'm not sure what are you referring here. I've just said that if we need to have one plugin depend on another, then doing this as part of the plugin manager is the right way to do it.

... but I would still like some method of deploying this shared library using our existing plugin management mechanisms

One way is using static libraries. With -fvisibility=hidden plus some other useful options the linker is able to remove the unused code.
Another option is using -rpath, but I'm not sure if it is available on windows or osx.

Another example is Tools+.

I've always felt that the tools+ should have been written over the original code and thus the old code removed. And this is the reason I'm not using the Tools+.
C::B has one very serious problem - it has tons of features that look like each other: tools/tools+, build-in search/threadsearch/incremental search.
We should try to do a better job in this regard - copy the good parts from all implementation and remove the bad once or at least extracting built-in tools and built-in search in plugins.

Curious to hear what others think. I won't spend any more time on this if enough of you think this is a dead end.

As I've said in the other topic: if you need this feature go on and implement it. Handling dependencies automatically is the only proper way to do this if you need it. Playing with the weights and so on is just a hack.

[beqroson]

Curious to hear what others think. I won't spend any more time on this if enough of you think this is a dead end.

I dont know much about the inner workings of Codeblocks. But it sounds useful to me. If one plugin does a few things on disc for example, like scanning a lot of header files. Instead of another plugin doing the same thing, it could get the result from the first plugin. If two plugins need the same mechanism that would not use amounts of resources, however, it would be better if each plugin manages on its own. Also, it would be necessary to select such behavior that does not really change over longer periods of time. If the interface of the plugin would change or the format of the data, then the interdependency could break too often for it to be usable. The best thing would be that the interdependency actually never breaks, which would demand that the plugin kept backwards compatible of its inner workings. Compare with DLL-hell.

[dmoore]

On a separate topic:
Have you measured the time needed to load C::B with and without your patch.
Currently C::B loads pretty slow if all contrib plugins are installed, but disabled.
Does you patch improve the load time?

Some, C::B still has to read all of the manifests, but no DLLs for disabled plugins get loaded. I will do some timings when I have time.

Some timings on an i386 laptop

Results for safemode

Code

time codeblocks --safe-mode -ns -ni --multiple-instance (quitting immediately after)
before: ~10.0s first time*, ~5.8s subsequently
after: ~2.8s first time*, ~2.6s subsequently

* NOT from cold start (so don't trust these numbers)

Results with only core plugins enabled

Code

time codeblocks -ns -ni --multiple-instance (quitting immediately after)
before: ~8.1s**
after: ~5.3s**

** based on second and subsequent runs

[dmoore]

Not what you said in the other thread...  

I'm not sure what are you referring here. I've just said that if we need to have one plugin depend on another, then doing this as part of the plugin manager is the right way to do it.

But how do you avoid linking one plugin against some subset of another?

... but I would still like some method of deploying this shared library using our existing plugin management mechanisms

One way is using static libraries. With -fvisibility=hidden plus some other useful options the linker is able to remove the unused code.
Another option is using -rpath, but I'm not sure if it is available on windows or osx.

Not sure what you mean, but probably because I don't sufficiently understand shared object linking. Suppose we have plugin A, plugin B and a library C that they both depend on. If I statically link A and C, how do I make sure B can find the stuff in C?

Another example is Tools+.

I've always felt that the tools+ should have been written over the original code and thus the old code removed. And this is the reason I'm not using the Tools+.
C::B has one very serious problem - it has tons of features that look like each other: tools/tools+, build-in search/threadsearch/incremental search.
We should try to do a better job in this regard - copy the good parts from all implementation and remove the bad once or at least extracting built-in tools and built-in search in plugins.

Agree, it's just a bit more work to make sure Tools+ properly duplicates the old features (and clean up the UI) then remove the old stuff from C::B core. (There are historical reasons why it is the way it is, but they don't matter here.) But even if we do that, to allow other plugins to use Tools+ features means putting some of that stuff in the SDK or allowing for some way for plugins to depend on one another.

Curious to hear what others think. I won't spend any more time on this if enough of you think this is a dead end.

As I've said in the other topic: if you need this feature go on and implement it. Handling dependencies automatically is the only proper way to do this if you need it. Playing with the weights and so on is just a hack.

I wasn't proposing weights, I was proposing an explicit dependency chain be specified in the manifest (i.e. a "dependency" attribute with a comma separated list of the unique names of the plugins that it depends on). I don't think you can get by with just trying to load the lib and praying it works.

[oBFusCATed]

Not sure what you mean, but probably because I don't sufficiently understand shared object linking. Suppose we have plugin A, plugin B and a library C that they both depend on. If I statically link A and C, how do I make sure B can find the stuff in C?

A static lib is just an archive with object files, so if you link both A and B to C (which is static lib), then both A and B will have all the code in C.
A and B will have their own copy of the code, nothing will be shared and they will have no external dependency to C. They will be self-sufficient.

In this case the data structures for A and B will be separate/duplicated in memory.

p.s. about the weights: I've mentioned it because it is another simpler approach to handle plugin deps.

[dmoore]

In this case the data structures for A and B will be separate/duplicated in memory.

That's a non-solution.

p.s. about the weights: I've mentioned it because it is another simpler approach to handle plugin deps.

But fundamentally broken. if you disable a plugin with weight >X then you would have to unload all plugins with weight < X to be sure you haven't just crippled (and made unstable) a dependent plugin.

Your philosophical objection to linking plugins together doesn't make a lot of sense to me. It sort of sounds like you are trying to sandbox plugins, but when plugins can link against any lib in the system it isn't really much of a sandbox. And it's not like our plugins expose the user to any more risk than C::B itself -- even scripts have access to wxExecute. So what am I missing? That it's just cleaner to link against a common lib (a la wxSmith) than linking plugins together?

A real problem is the technical one i.e. the need to use -rpath (or, on windows, AddDllDirectory). The former only lets you set a link directory relative to $ORIGIN, which would make it impossible to put objects that a plugin depends on in users home.

I guess I can live with using the approach we use for wxSmith and putting all of the shared stuff in a separate lib in the same dir as libcodeblocks (or some other dir that we can -rpath to). It makes a deployment a bit more of a pain unless the core plugin is part of contrib because we don't have a mechanism for installing the shared libs from a cbplugin, which, at least on linux, requires admin rights. But maybe that's what I/we should work on?

[oBFusCATed]

Hm, I think I'm being misunderstood. I don't like the idea of having plugins for plugins, I'm relatively ok with the idea of plugin communications.
So, lets stop arguing and just do it if you need it.

Have you though of putting all of your python subplugins in a single binary? I think this is possible already in C::B and probably will be easier for the user.

[frithjofh]

maybe not plugins for plugins.

maybe some base functionality inside c::b like services. these provide interface and API. the work internally with plugins that implement and parametrize the service.

then, outside, plugins that use the services through their public interface.

that way the plugins wont be depending on each other. only outside plugins on services.

maybe some functionalities of c::b are already so general, that actually no one ever unloads the plugin representing that functionality. maybe some things have been plugins for so long that nobody questions if they really should be plugins... or at least plugins directly exposed to the user. defintely if one looks at the plugins today, there are already two groups of them: those that one virtually HAS to use in order to do anything with an IDE, and then all the others...

for instance an IDE would likely always offer some compiler. well, that compiler would be a service, the service to be able to compile something. inside the service there would be plugins to specialize the service, for instance being of type gcc or intel compiler, local or distant, etc.

AST would be a general service specialized by plugin for each language. it would be used outside by plugins like CC or refactoring or searching or statistics or symbol browser or whatever.

just thinking. most probably this would be so great a change it is not worth while. not to repeat the netscape experience... by all means.