XML based compilers

[Alpha]

It seems there were some plans at one point to convert the compiler system from hard-coded to xml.  The attached patch gives an example of a possible way this could be done.  Before I spend time converting the rest of the compilers, I was wondering what comments or suggestions anyone had on method with which I attempted this (and to check how much interest there currently is in xml based compilers).

This patch only moves the compiler options into the xml file (no regexps and no auto-detect), however I think I could create an additional function to load (custom) pure xml compilers (which would be fully functional except for lacking auto-detect).

[MortenMacFly]

Indeed - this was in internal task, just we didn't have resources so far. This was/is the internal planing of the dev team:

Task:
- Make compiler options more flexible

The compiler options are added like:

Code

m_Options.AddOption(description, unique_compiler_key, category);

• Make this available through an XML settings file
• Design/implement an UI to easily edit these options
• Probably allow to derive form an other's compiler settings

If you did a first step that would be very nice. I'll certainly look into it - as this would be a #1 priority change. Expect feedback... just my time is very limited atm...

[MortenMacFly]

Ok - I finally had a look into it. It goes even further then what I had in mind concerning the options (which is good ).

Some notes:

• You/we should consider a simple UI in the end where the user can adjust these options as desired so they become really flexible. Just keep in mind if designing interfaces for example.
• I would have preferred to see using TinyXML the persistence layer as we do it with any other XML based options files, but this might not be so important (I recall when we decided to go for TinyXML the XML layer in wxWidgets was not that powerful.)
• We should also consider to inform "darmar" (the Fortran project maintainer)... I'll do this via PM.
• We need to update the "update" scripts accordingly, too

From my side you clearly got the "go", knowing that this is an awful lot of work. But it would be one of the most welcome contributions currently!

[Alpha]

Conversion process started.  I wrote a small text manipulation plugin (which I will release later) to help me, so it might be a bit less work than first estimated.

• You/we should consider a simple UI in the end where the user can adjust these options as desired so they become really flexible. Just keep in mind if designing interfaces for example.

My current idea is to have a right-click menu with:

Code

New flag...
Modify flag...
Delete flag

I have not tried creating it yet (I am working on converting the compilers first).

• I would have preferred to see using TinyXML the persistence layer as we do it with any other XML based options files, but this might not be so important (I recall when we decided to go for TinyXML the XML layer in wxWidgets was not that powerful.)

I had selected wxXml because it had enough features for my purposes, and so that I did not have to worry about converting wxStrings.  Is it fine to leave as-is, or should I switch to TinyXML?

[MortenMacFly]

My current idea is to have a right-click menu with:

Code

New flag...
Modify flag...
Delete flag

That's an easy way for a first step, sure.

I had selected wxXml because it had enough features for my purposes, and so that I did not have to worry about converting wxStrings.  Is it fine to leave as-is, or should I switch to TinyXML?

Well as I said - wxXml has changes a lot, so if you have all functions you need, I'm fine with it.

[Alpha]

Attached is the initial conversion of all of the compilers.  There is a decent chance (with so much text to convert) that I have missed/messed up something, so I would appreciate usability reports from anyone with time to test this.

My next step is to add user customization features.

[darmar]

Thank you Alpha for your efforts.

I would like to add what I am missing in the current C::B implementation of compilers:
Some of compilers require options to be specified in a linker stage. But there is no possibility to have "Linker Flags" just like "Compiler Flags". Currently users can add required options in "Other linker options" window, but this is not as convenient as with flags.

Maybe it could be possible to add "Linker Flags" for "XML based compilers"?

[thomas]

(Some of) the reasons why the XML idea was abandoned back then were

• XML is actually rather ill-suited for the purpose (scripting is a much better fit). Mea culpa. At that time I made the same mistake that many people make, I saw XML as great for everything. It isn't. It's great for some things, but not for all.
• It goes far beyond "some compiler switches"
  • an abstraction of functionality would be needed, this needs to be properly planned and documented first, and it had better be complete (once set up, it's a pain to change again!)
  • Some flags/functions are mutually exclusive on some compilers, sometimes depending on a third one, and sometimes a flag needs to be passed to another stage or requires another one being passed on a different stage
  • compiler and debugger are a kind-of union
  • error/warning parsing is tightly bound to the compiler as well
• The build process would have to be reworked (and probably project files as well)
• Anything else would be the same as we already have, just with a new file format
• The two people mainly involved in the planning of that beast found themselves out of time doing all that (or doing much any more either way)
• Reworking projects would be a compatibility issue, so a backwards-compatibility layer would have to be written and maintained
• Development would have to take place in a "secret branch" or we'd drown in a flood of complaints from people using nightly builds. Someone would need to merge the branch later (which out of experience I can tell is painful and takes days of work).

That said, if someone is still willing to work on this beast, great. But please do it properly, don't just tack XML onto the existing system, which frankly would be just shit (no offense intended).

[Alpha]

(Warning: long post.)

Maybe it could be possible to add "Linker Flags" for "XML based compilers"?

This is sort of already available in the current system:

Code

void AddOption(const wxString& name,
               const wxString& option,
               const wxString& category = _("General"),
               const wxString& additionalLibs = wxEmptyString,
               bool doChecks = false,
               const wxString& checkAgainst = wxEmptyString,
               const wxString& checkMessage = wxEmptyString);

Anything in additionalLibs is sent to the linker.  So in this XML system:

Code

<Option name="Strip all symbols from binary (minimizes size)"
        additionalLibs="-s"/>

However, this can lead to confusion because it does not explicitly mention that compiler flags can control the linker.  I am considering the possibility of a "Linker Flags" window, but I am not sure if there are enough common features to warrant the extra space usage (as the functionality is, in essence, already available).

• Some flags/functions are mutually exclusive on some compilers, sometimes depending on a third one, and sometimes a flag needs to be passed to another stage or requires another one being passed on a different stage

I am afraid your logic in this statement is to complicated for me to follow.  If Code::Blocks current compiler system can do these, then this XML system can as well.  Compilers are currently not completely decoupled into XML; only the contents of the Reset() method: m_Programs, m_Switches, m_Options, and m_Commands.  These items are loaded from the compiler's XML options file (instead of being hard-coded).

• The build process would have to be reworked (and probably project files as well)
• Anything else would be the same as we already have, just with a new file format

All changes are currently highly localized; the compilers' interfaces function exactly the same, the only difference is how they populate their options (pure C++ compilers are still completely valid as well).
That said, the compilers I have converted are now a hybrid of C++ (for regexps and auto-detect) and XML (for options/flags).  My final (planned) step would be to write a compilerXML.cpp which would load any pure XML compilers that had been detected during start-up.  (No compilers are, as of yet, dynamically loaded.)

[...]
don't just tack XML onto the existing system
[...]

This may have been exactly what I have done.  However, I think it still holds the advantage of providing a framework to make compiler flags easily configurable.  Have you had a chance to explore my implementation? - I know I am not an experienced programmer, so I would appreciate any critique on if this is even the "right" method to move forwards.


Would anyone like me to write up documentation for the current XML format?  (Otherwise I will wait a bit until I am reasonably sure that it is set.)

[Alpha]

This is not exactly related; in exploring the compilers, I found that the variables

Code

bool doChecks
const wxString& checkAgainst
const wxString& checkMessage

have no effect.  I assume that they were planned, but no one finished the implementation.  If I were to implement this, should I bundle it with my XML compiler patch, or should I make it directly against the trunk?

Edit: Never mind; against trunk.

[MortenMacFly]

That said, if someone is still willing to work on this beast, great. But please do it properly, don't just tack XML onto the existing system, which frankly would be just shit (no offense intended).

Well Thomas, I got you point, but please: Don't request a re-build of the core from a single person involved just recently.

And I clearly disagree with you here: What I think completely sucks is that these options are hard-coded inside each compiler. So we basically need to re-build everything if just a single option changes. Also, some compilers are basically the same just they differ in some options. So you could easily go one step further (as Alpha suggested already) and have a "generic" compiler that loads and configures itself with such configuration. This will definitely  work so some extend.
Also, if you change the advanced options of the compiler they are already XML based, just inside the C::B config where they not belong to IMHO. Thus, making all options XML based is just consequent.

So: I agree that there may be the ultimate solution possible, but we don't have time and resources for it ATM. Nevertheless the approach proposed here is a huge step in the right direction, clearly, so we should go for it. Furthermore I don't think the script based approach will be a good idea at the moment. The scripting engine is the show-stopper #1 for the 64 bit build of C::B as SQPlus was abandoned and we have to maintain it ourself. But this is such a low-level stuff that you need an own team of devs just for that. already, meanwhile you cannot even update the underlying squirrel anymore due the massively out-dated SQPlus layer. If we make everything script based we end-up nowhere sooner or later, IMHO.

[thomas]

Well, if nothing helps, we can still move back to Angelscript, which is well supported. Ironically, the reason why we abandoned Angelscript back then was that it didn't do 64 bits at that time... 

The thing about scripting is you could not only concatenate some strings, but also do logic inside them. In fact, not only the composition of the options dialog, but the entire build process could be a script which calls some native SDK functions (one would need a well-defined interface, of course). Not much unlike the project wizards (yep, those that I hate so much) work. What I'd wish for is nothing else but the "generic compiler" that you talked about earlier. Except with scripting it would certainly work, as a script can trivially do "just anything", and with XML I see the task of making something that works for all as rather difficult.

Now XML is mighty fine for configuring a few strings, but the logic is still all hardcoded, and the way compiler options are handled is poor at best. You know I'm no friend of "too much darn smartness", but compiler options in Code::Blocks aren't smart at all. You can for example select "do not optimize" together with all levels of optimization, all at the same time (same goes for warnings), you can select a dozen architectures at the same time, and you can generate debug info and strip at the same time. Certainly you can expect a programmer to somewhat know what he's doing, but how often does it happen to you that you accidentially strip your debug info and wonder why the debugger doesn't come up (if it doesn't ever happen to you, you're provably more intelligent than I am, I have this about once every other week). In any case, it wouldn't hurt if the user interface was a bit clearer. Alas, this is darn hard to hardcode.

What I'm fearing is that if this kind of works (even if it works only just a little bit better than what we have now) it will stay like this for the next decade. 

If only I had the time...

[Alpha]

You know I'm no friend of "too much darn smartness", but compiler options in Code::Blocks aren't smart at all. You can for example select "do not optimize" together with all levels of optimization, all at the same time (same goes for warnings), you can select a dozen architectures at the same time, and you can generate debug info and strip at the same time. Certainly you can expect a programmer to somewhat know what he's doing, but how often does it happen to you that you accidentially strip your debug info and wonder why the debugger doesn't come up (if it doesn't ever happen to you, you're provably more intelligent than I am, I have this about once every other week). In any case, it wouldn't hurt if the user interface was a bit clearer. Alas, this is darn hard to hardcode.

Now that you mention it, I had already created the ability to fix this in my local version with:

Code

struct CompOption
{
	// following comments are an example of an option
	wxString name;			// "Profile code"
	wxString option;		// "-pg"
	wxString additionalLibs;// "-lgmon"
	bool enabled;			// true/false
	wxString category;		// "Profiling"
	bool doChecks;			// true/false
	wxString checkAgainst;	// "-O -O1 -O2 -O3 -Os" (will check for these options and if any of them is found, will display the following message)
	wxString checkMessage;	// "You have optimizations enabled. This is Not A Good Thing(tm) when producing debugging symbols..."
	wxString supersedes;    // "-O -O1 -O2" (will check for these options and disable any of them that are found)
	bool exclusive;         // true/false (will ensure that only one item in this category is ever selected)
};

and

Code

void CompilerOptionsDlg::OnOptionToggled(wxCommandEvent& event)
{
    wxCheckListBox* list = XRCCTRL(*this, "lstCompilerOptions", wxCheckListBox);
    int sel = event.GetInt();
    CompOption* copt = m_Options.GetOptionByName(list->GetString(sel));
    if (copt)
    {
        copt->enabled = list->IsChecked(sel);
        if (copt->enabled)
        {
            if (copt->doChecks)
            {
                wxArrayString check = GetArrayFromString(copt->checkAgainst, _T(" "));
                for (size_t i = 0; i < check.Count(); i++)
                {
                    CompOption* against = m_Options.GetOptionByOption(check[i]);
                    if (against && against->enabled)
                    {
                        AnnoyingDialog dlg(_("Compiler options conflict"),
                                           copt->checkMessage,
                                           wxART_INFORMATION,
                                           AnnoyingDialog::OK,
                                           wxID_OK);
                        dlg.ShowModal();
                        break;
                    }
                }
            }
            if (copt->supersedes != wxEmptyString)
            {
                wxArrayString supersede = GetArrayFromString(copt->supersedes, _T(" "));
                for (size_t i = 0; i < supersede.Count(); i++)
                {
                    for (size_t j = 0; j < m_Options.GetCount(); j++)
                    {
                        if (copt != m_Options.GetOption(j) &&
                            supersede[i] == m_Options.GetOption(j)->option)
                        {
                            list->Check(j, false);
                            m_Options.GetOption(j)->enabled = false;
                        }
                    }
                }
            }
            if (copt->exclusive)
            {
                for (size_t i = 0; i < m_Options.GetCount(); i++)
                {
                    if (copt != m_Options.GetOption(i) &&
                        copt->category == m_Options.GetOption(i)->category)
                    {
                        list->Check(i, false);
                        m_Options.GetOption(i)->enabled = false;
                    }
                }
            }
        }
    }
    m_bDirty = true;
} // OnOptionToggled

(I implemented doChecks and added supersedes and exclusive.  I did not update the current compilers to take advantage of this functionality, however my quick test with the XML compilers worked beautifully - the next patch I upload will contain these and (hopefully) dynamically configurable flags.)

What I am not currently sure about is if it is better (and in which cases) to show a warning dialog, and when to just automatically check/uncheck the conflicting options.

Well, if nothing helps, we can still move back to Angelscript, which is well supported. Ironically, the reason why we abandoned Angelscript back then was that it didn't do 64 bits at that time... 

With respects to scripting, I am curious if anyone has ever discussed the pros/cons of Python?

[oBFusCATed]

With respects to scripting, I am curious if anyone has ever discussed the pros/cons of Python?

Python is not made for embedding, it is their agenda that the applications should be written in python, not extended by python.
It is doable but the API is pretty low level or we need to require Boost.Python, which is mighty heavy.
Also shipping python on windows will be problematic.

In my opinion current scripting in C::B is pretty bad, because SQPlus is mighty stupid. I've tried to add some scripting for the debugger and failed quite miserably
Also the docs for squirrel are not helping much, as they are written for someone knowing what he is doing, not for someone who is new to it and scripting.

[MortenMacFly]

In my opinion current scripting in C::B is pretty bad, because SQPlus is mighty stupid. I've tried to add some scripting for the debugger and failed quite miserably

We are becoming slightly off-topic, however...:
I think Scrat (http://scrat.sourceforge.net/) would be another option for a replacement of SQPlus. But as Thomas said, if AngelScript is still alive, actively developed (and supported) and supports 64 bit we might have done a wrong choice the time back...