Author Topic: XML based compilers  (Read 95170 times)

Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1506
XML based compilers
« on: June 15, 2012, 01:00:36 pm »
It seems there were some plans at one point to convert the compiler system from hard-coded to xml.  The attached patch gives an example of a possible way this could be done.  Before I spend time converting the rest of the compilers, I was wondering what comments or suggestions anyone had on method with which I attempted this (and to check how much interest there currently is in xml based compilers).

This patch only moves the compiler options into the xml file (no regexps and no auto-detect), however I think I could create an additional function to load (custom) pure xml compilers (which would be fully functional except for lacking auto-detect).

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9415
Re: XML based compilers
« Reply #1 on: June 15, 2012, 02:39:42 pm »
Indeed - this was in internal task, just we didn't have resources so far. This was/is the internal planing of the dev team:

Quote
Task:
- Make compiler options more flexible

The compiler options are added like:
Code: [Select]
m_Options.AddOption(description, unique_compiler_key, category);
  • Make this available through an XML settings file
  • Design/implement an UI to easily edit these options
  • Probably allow to derive form an other's compiler settings

If you did a first step that would be very nice. I'll certainly look into it - as this would be a #1 priority change. Expect feedback... just my time is very limited atm...
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9415
Re: XML based compilers
« Reply #2 on: June 16, 2012, 09:22:46 am »
Ok - I finally had a look into it. It goes even further then what I had in mind concerning the options (which is good ;)).

Some notes:
  • You/we should consider a simple UI in the end where the user can adjust these options as desired so they become really flexible. Just keep in mind if designing interfaces for example.
  • I would have preferred to see using TinyXML the persistence layer as we do it with any other XML based options files, but this might not be so important (I recall when we decided to go for TinyXML the XML layer in wxWidgets was not that powerful.)
  • We should also consider to inform "darmar" (the Fortran project maintainer)... I'll do this via PM.
  • We need to update the "update" scripts accordingly, too

From my side you clearly got the "go", knowing that this is an awful lot of work. But it would be one of the most welcome contributions currently! 8)
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1506
Re: XML based compilers
« Reply #3 on: June 16, 2012, 06:06:35 pm »
Conversion process started.  I wrote a small text manipulation plugin (which I will release later) to help me, so it might be a bit less work than first estimated.

  • You/we should consider a simple UI in the end where the user can adjust these options as desired so they become really flexible. Just keep in mind if designing interfaces for example.
My current idea is to have a right-click menu with:
Code: [Select]
New flag...
Modify flag...
Delete flag
I have not tried creating it yet (I am working on converting the compilers first).

  • I would have preferred to see using TinyXML the persistence layer as we do it with any other XML based options files, but this might not be so important (I recall when we decided to go for TinyXML the XML layer in wxWidgets was not that powerful.)
I had selected wxXml because it had enough features for my purposes, and so that I did not have to worry about converting wxStrings.  Is it fine to leave as-is, or should I switch to TinyXML?

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9415
Re: XML based compilers
« Reply #4 on: June 17, 2012, 12:28:05 pm »
My current idea is to have a right-click menu with:
Code: [Select]
New flag...
Modify flag...
Delete flag
That's an easy way for a first step, sure.

I had selected wxXml because it had enough features for my purposes, and so that I did not have to worry about converting wxStrings.  Is it fine to leave as-is, or should I switch to TinyXML?
Well as I said - wxXml has changes a lot, so if you have all functions you need, I'm fine with it.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1506
Re: XML based compilers
« Reply #5 on: June 18, 2012, 02:38:12 am »
Attached is the initial conversion of all of the compilers.  There is a decent chance (with so much text to convert) that I have missed/messed up something, so I would appreciate usability reports from anyone with time to test this.

My next step is to add user customization features.

Offline darmar

  • Multiple posting newcomer
  • *
  • Posts: 39
Re: XML based compilers
« Reply #6 on: June 18, 2012, 09:05:43 am »
Thank you Alpha for your efforts.

I would like to add what I am missing in the current C::B implementation of compilers:
Some of compilers require options to be specified in a linker stage. But there is no possibility to have "Linker Flags" just like "Compiler Flags". Currently users can add required options in "Other linker options" window, but this is not as convenient as with flags.

Maybe it could be possible to add "Linker Flags" for "XML based compilers"?

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: XML based compilers
« Reply #7 on: June 18, 2012, 12:28:36 pm »
(Some of) the reasons why the XML idea was abandoned back then were
  • XML is actually rather ill-suited for the purpose (scripting is a much better fit). Mea culpa. At that time I made the same mistake that many people make, I saw XML as great for everything. It isn't. It's great for some things, but not for all.
  • It goes far beyond "some compiler switches"
    • an abstraction of functionality would be needed, this needs to be properly planned and documented first, and it had better be complete (once set up, it's a pain to change again!)
    • Some flags/functions are mutually exclusive on some compilers, sometimes depending on a third one, and sometimes a flag needs to be passed to another stage or requires another one being passed on a different stage
    • compiler and debugger are a kind-of union
    • error/warning parsing is tightly bound to the compiler as well
  • The build process would have to be reworked (and probably project files as well)
  • Anything else would be the same as we already have, just with a new file format
  • The two people mainly involved in the planning of that beast found themselves out of time doing all that (or doing much any more either way)
  • Reworking projects would be a compatibility issue, so a backwards-compatibility layer would have to be written and maintained
  • Development would have to take place in a "secret branch" or we'd drown in a flood of complaints from people using nightly builds. Someone would need to merge the branch later (which out of experience I can tell is painful and takes days of work).

That said, if someone is still willing to work on this beast, great. But please do it properly, don't just tack XML onto the existing system, which frankly would be just shit (no offense intended).
« Last Edit: June 18, 2012, 12:30:13 pm by thomas »
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1506
Re: XML based compilers
« Reply #8 on: June 18, 2012, 03:20:55 pm »
(Warning: long post.)

Maybe it could be possible to add "Linker Flags" for "XML based compilers"?
This is sort of already available in the current system:
Code: [Select]
void AddOption(const wxString& name,
               const wxString& option,
               const wxString& category = _("General"),
               const wxString& additionalLibs = wxEmptyString,
               bool doChecks = false,
               const wxString& checkAgainst = wxEmptyString,
               const wxString& checkMessage = wxEmptyString);
Anything in additionalLibs is sent to the linker.  So in this XML system:
Code: [Select]
<Option name="Strip all symbols from binary (minimizes size)"
        additionalLibs="-s"/>
However, this can lead to confusion because it does not explicitly mention that compiler flags can control the linker.  I am considering the possibility of a "Linker Flags" window, but I am not sure if there are enough common features to warrant the extra space usage (as the functionality is, in essence, already available).

  • Some flags/functions are mutually exclusive on some compilers, sometimes depending on a third one, and sometimes a flag needs to be passed to another stage or requires another one being passed on a different stage
I am afraid your logic in this statement is to complicated for me to follow.  If Code::Blocks current compiler system can do these, then this XML system can as well.  Compilers are currently not completely decoupled into XML; only the contents of the Reset() method: m_Programs, m_Switches, m_Options, and m_Commands.  These items are loaded from the compiler's XML options file (instead of being hard-coded).

  • The build process would have to be reworked (and probably project files as well)
  • Anything else would be the same as we already have, just with a new file format
All changes are currently highly localized; the compilers' interfaces function exactly the same, the only difference is how they populate their options (pure C++ compilers are still completely valid as well).
That said, the compilers I have converted are now a hybrid of C++ (for regexps and auto-detect) and XML (for options/flags).  My final (planned) step would be to write a compilerXML.cpp which would load any pure XML compilers that had been detected during start-up.  (No compilers are, as of yet, dynamically loaded.)

[...]
don't just tack XML onto the existing system
[...]
This may have been exactly what I have done.  However, I think it still holds the advantage of providing a framework to make compiler flags easily configurable.  Have you had a chance to explore my implementation? - I know I am not an experienced programmer, so I would appreciate any critique on if this is even the "right" method to move forwards.


Would anyone like me to write up documentation for the current XML format?  (Otherwise I will wait a bit until I am reasonably sure that it is set.)

Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1506
Re: XML based compilers
« Reply #9 on: June 18, 2012, 03:34:26 pm »
This is not exactly related; in exploring the compilers, I found that the variables
Code: [Select]
bool doChecks
const wxString& checkAgainst
const wxString& checkMessage
have no effect.  I assume that they were planned, but no one finished the implementation.  If I were to implement this, should I bundle it with my XML compiler patch, or should I make it directly against the trunk?

Edit: Never mind; against trunk.
« Last Edit: June 18, 2012, 04:34:00 pm by Alpha »

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9415
Re: XML based compilers
« Reply #10 on: June 18, 2012, 08:17:21 pm »
That said, if someone is still willing to work on this beast, great. But please do it properly, don't just tack XML onto the existing system, which frankly would be just shit (no offense intended).
Well Thomas, I got you point, but please: Don't request a re-build of the core from a single person involved just recently.

And I clearly disagree with you here: What I think completely sucks is that these options are hard-coded inside each compiler. So we basically need to re-build everything if just a single option changes. Also, some compilers are basically the same just they differ in some options. So you could easily go one step further (as Alpha suggested already) and have a "generic" compiler that loads and configures itself with such configuration. This will definitely  work so some extend.
Also, if you change the advanced options of the compiler they are already XML based, just inside the C::B config where they not belong to IMHO. Thus, making all options XML based is just consequent.

So: I agree that there may be the ultimate solution possible, but we don't have time and resources for it ATM. Nevertheless the approach proposed here is a huge step in the right direction, clearly, so we should go for it. Furthermore I don't think the script based approach will be a good idea at the moment. The scripting engine is the show-stopper #1 for the 64 bit build of C::B as SQPlus was abandoned and we have to maintain it ourself. But this is such a low-level stuff that you need an own team of devs just for that. already, meanwhile you cannot even update the underlying squirrel anymore due the massively out-dated SQPlus layer. If we make everything script based we end-up nowhere sooner or later, IMHO.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: XML based compilers
« Reply #11 on: June 19, 2012, 02:05:30 am »
Well, if nothing helps, we can still move back to Angelscript, which is well supported. Ironically, the reason why we abandoned Angelscript back then was that it didn't do 64 bits at that time...  ;D

The thing about scripting is you could not only concatenate some strings, but also do logic inside them. In fact, not only the composition of the options dialog, but the entire build process could be a script which calls some native SDK functions (one would need a well-defined interface, of course). Not much unlike the project wizards (yep, those that I hate so much) work. What I'd wish for is nothing else but the "generic compiler" that you talked about earlier. Except with scripting it would certainly work, as a script can trivially do "just anything", and with XML I see the task of making something that works for all as rather difficult.

Now XML is mighty fine for configuring a few strings, but the logic is still all hardcoded, and the way compiler options are handled is poor at best. You know I'm no friend of "too much darn smartness", but compiler options in Code::Blocks aren't smart at all. You can for example select "do not optimize" together with all levels of optimization, all at the same time (same goes for warnings), you can select a dozen architectures at the same time, and you can generate debug info and strip at the same time. Certainly you can expect a programmer to somewhat know what he's doing, but how often does it happen to you that you accidentially strip your debug info and wonder why the debugger doesn't come up (if it doesn't ever happen to you, you're provably more intelligent than I am, I have this about once every other week). In any case, it wouldn't hurt if the user interface was a bit clearer. Alas, this is darn hard to hardcode.

What I'm fearing is that if this kind of works (even if it works only just a little bit better than what we have now) it will stay like this for the next decade.  ::)

If only I had the time... :(
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Alpha

  • Developer
  • Lives here!
  • *****
  • Posts: 1506
Re: XML based compilers
« Reply #12 on: June 19, 2012, 04:19:21 am »
You know I'm no friend of "too much darn smartness", but compiler options in Code::Blocks aren't smart at all. You can for example select "do not optimize" together with all levels of optimization, all at the same time (same goes for warnings), you can select a dozen architectures at the same time, and you can generate debug info and strip at the same time. Certainly you can expect a programmer to somewhat know what he's doing, but how often does it happen to you that you accidentially strip your debug info and wonder why the debugger doesn't come up (if it doesn't ever happen to you, you're provably more intelligent than I am, I have this about once every other week). In any case, it wouldn't hurt if the user interface was a bit clearer. Alas, this is darn hard to hardcode.
Now that you mention it, I had already created the ability to fix this in my local version with:
Code: [Select]
struct CompOption
{
// following comments are an example of an option
wxString name; // "Profile code"
wxString option; // "-pg"
wxString additionalLibs;// "-lgmon"
bool enabled; // true/false
wxString category; // "Profiling"
bool doChecks; // true/false
wxString checkAgainst; // "-O -O1 -O2 -O3 -Os" (will check for these options and if any of them is found, will display the following message)
wxString checkMessage; // "You have optimizations enabled. This is Not A Good Thing(tm) when producing debugging symbols..."
wxString supersedes;    // "-O -O1 -O2" (will check for these options and disable any of them that are found)
bool exclusive;         // true/false (will ensure that only one item in this category is ever selected)
};
and
Code: [Select]
void CompilerOptionsDlg::OnOptionToggled(wxCommandEvent& event)
{
    wxCheckListBox* list = XRCCTRL(*this, "lstCompilerOptions", wxCheckListBox);
    int sel = event.GetInt();
    CompOption* copt = m_Options.GetOptionByName(list->GetString(sel));
    if (copt)
    {
        copt->enabled = list->IsChecked(sel);
        if (copt->enabled)
        {
            if (copt->doChecks)
            {
                wxArrayString check = GetArrayFromString(copt->checkAgainst, _T(" "));
                for (size_t i = 0; i < check.Count(); i++)
                {
                    CompOption* against = m_Options.GetOptionByOption(check[i]);
                    if (against && against->enabled)
                    {
                        AnnoyingDialog dlg(_("Compiler options conflict"),
                                           copt->checkMessage,
                                           wxART_INFORMATION,
                                           AnnoyingDialog::OK,
                                           wxID_OK);
                        dlg.ShowModal();
                        break;
                    }
                }
            }
            if (copt->supersedes != wxEmptyString)
            {
                wxArrayString supersede = GetArrayFromString(copt->supersedes, _T(" "));
                for (size_t i = 0; i < supersede.Count(); i++)
                {
                    for (size_t j = 0; j < m_Options.GetCount(); j++)
                    {
                        if (copt != m_Options.GetOption(j) &&
                            supersede[i] == m_Options.GetOption(j)->option)
                        {
                            list->Check(j, false);
                            m_Options.GetOption(j)->enabled = false;
                        }
                    }
                }
            }
            if (copt->exclusive)
            {
                for (size_t i = 0; i < m_Options.GetCount(); i++)
                {
                    if (copt != m_Options.GetOption(i) &&
                        copt->category == m_Options.GetOption(i)->category)
                    {
                        list->Check(i, false);
                        m_Options.GetOption(i)->enabled = false;
                    }
                }
            }
        }
    }
    m_bDirty = true;
} // OnOptionToggled
(I implemented doChecks and added supersedes and exclusive.  I did not update the current compilers to take advantage of this functionality, however my quick test with the XML compilers worked beautifully - the next patch I upload will contain these and (hopefully) dynamically configurable flags.)

What I am not currently sure about is if it is better (and in which cases) to show a warning dialog, and when to just automatically check/uncheck the conflicting options.

Well, if nothing helps, we can still move back to Angelscript, which is well supported. Ironically, the reason why we abandoned Angelscript back then was that it didn't do 64 bits at that time...  ;D
With respects to scripting, I am curious if anyone has ever discussed the pros/cons of Python?

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 9870
Re: XML based compilers
« Reply #13 on: June 19, 2012, 10:14:47 am »
With respects to scripting, I am curious if anyone has ever discussed the pros/cons of Python?
Python is not made for embedding, it is their agenda that the applications should be written in python, not extended by python.
It is doable but the API is pretty low level or we need to require Boost.Python, which is mighty heavy.
Also shipping python on windows will be problematic.

In my opinion current scripting in C::B is pretty bad, because SQPlus is mighty stupid. I've tried to add some scripting for the debugger and failed quite miserably :(
Also the docs for squirrel are not helping much, as they are written for someone knowing what he is doing, not for someone who is new to it and scripting.
<debugger plugin maintainer>
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9415
Re: XML based compilers
« Reply #14 on: June 19, 2012, 11:05:25 am »
In my opinion current scripting in C::B is pretty bad, because SQPlus is mighty stupid. I've tried to add some scripting for the debugger and failed quite miserably :(
We are becoming slightly off-topic, however...:
I think Scrat (http://scrat.sourceforge.net/) would be another option for a replacement of SQPlus. But as Thomas said, if AngelScript is still alive, actively developed (and supported) and supports 64 bit we might have done a wrong choice the time back... :-\
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ