Author Topic: Enhancements to the CB Build System  (Read 22599 times)

Offline scarphin

  • Lives here!
  • ****
  • Posts: 644
Enhancements to the CB Build System
« on: February 25, 2015, 03:00:55 pm »
The main purpose of these enhancements is to provide native Qt support with little or no user intervention within CB build system. Before going into discussion I want to make sure that none of these enhancements are specific to Qt, they are just improvements of current functionality which can be used for any other tool or framework. I'll try to submit further patches according to the directions I get here.

1- This patch provides an automatic compilation priority (weight) setting for new files added to the project. User Interfaces '.ui' in Qt (similar to .xrc files of wxwidgets) need to be compiled before the including header needs to access them and this patch saves the user to set each '.ui' file's weight manually. Patch here -> https://sourceforge.net/p/codeblocks/tickets/140/

2- This patch allows macros like $(project_file) etc... in the 'generated files' input in 'advanced compiler options' dialogue so generated intermediate files can go into their own folders like 'project_dir/generated_files/...'. Although my tests on Win7 and XP doesn't yield a problem, I'm not sure with this patch as my implementation is simpler than the current implementation and I suspect there must be a reason why it is not implemented like that in the first place. So please comment. Patch here -> https://sourceforge.net/p/codeblocks/tickets/141/

Offline LETARTARE

  • Lives here!
  • ****
  • Posts: 531
  • L'ami de l'homme.The friend of man.
    • LETARTARE
Re: Enhancements to the CB Build System
« Reply #1 on: February 25, 2015, 05:12:23 pm »
Hello,
do you have any examples of how to test your patches ?
Regards
CB-13483, plugins-sdk-2.25.0 : Collector-2.0.0, AddOnForQt-3.9.1
1-Win7 Business Pack1 64bits : wx-3.2.4, gcc-8.1.0,
2-OpenSuse::Leap-15.4-64bits : wx-3.2.4;gtk3, gcc-8.2.1,
=> !! The messages are translated by Deepl

Offline scarphin

  • Lives here!
  • ****
  • Posts: 644
Re: Enhancements to the CB Build System
« Reply #2 on: February 25, 2015, 06:05:44 pm »
Sure,

1- Create a new Qt project,
2- Set Qt directory and/or other compiler and/or linker options if needed,
3- Create a folder named 'gen' under projects directory,
4- Create a new '.ui' extension in 'compiler advanced options' for project's compiler,
5- Insert
Code
path_to_qt_bin\uic.exe $file -o $(project_dir)gen\ui_$file_name.h
into 'command line macro' with correct path,
6- Insert
Code
$(project_dir)gen\ui_$file_name.h
into 'generated files',
7- Set 'weight' to some value below 50,
8- Build the project.

As a result (if I haven't missed anything) a new header file under 'gen' directory will be generated before compilation starts. I'm attaching a zipped sample project with an already added '.ui' file which needs a 'qt' global variable to be set to build correctly. Settings in the 'advanced compiler options' need to be manually entered though. And the compiler for the project may need to set again correctly.
« Last Edit: February 25, 2015, 06:07:30 pm by scarphin »

Offline LETARTARE

  • Lives here!
  • ****
  • Posts: 531
  • L'ami de l'homme.The friend of man.
    • LETARTARE
Re: Enhancements to the CB Build System
« Reply #3 on: February 26, 2015, 10:58:43 am »
Hello,
thank you for the example, it works.
I tested with 'svn10127' and 'compiler1.patch'
I used the command line
Quote
$(#qt47)\bin\uic.exe $file -o $(project_dir)gen\ui_$file_name.h
See the attached picture.

1- create the directory 'gen' automatically,
2- each 'Build' is reconstructed systematically,
3- no trace of the pre-generation line,
4- the generated files are not stored in the project,

I think we can use the same approach for 'namefile.qrc -> 'qrc_namefile.cpp': nevertheless we must each file manually intervene.

Remains the most complicated, files containing macros to manage signals and slots.

Faced with this difficulty, I used the last two years a plugin script that does all the work.
I have proposed a binary version. See the discussion http://forums.codeblocks.org/index.php/topic,20000.msg136816.html#msg136816
You could try it and give me your results.
Best regards

« Last Edit: February 26, 2015, 11:30:43 am by LETARTARE »
CB-13483, plugins-sdk-2.25.0 : Collector-2.0.0, AddOnForQt-3.9.1
1-Win7 Business Pack1 64bits : wx-3.2.4, gcc-8.1.0,
2-OpenSuse::Leap-15.4-64bits : wx-3.2.4;gtk3, gcc-8.2.1,
=> !! The messages are translated by Deepl

Offline scarphin

  • Lives here!
  • ****
  • Posts: 644
Re: Enhancements to the CB Build System
« Reply #4 on: February 26, 2015, 12:20:13 pm »
Thanks for testing...

1- create the directory 'gen' automatically,
2- each 'Build' is reconstructed systematically,
3- no trace of the pre-generation line,
4- the generated files are not stored in the project,
I'm assuming these are what you think should be. If so:
1- That would be nice but pre/post-build steps are also designed that way and it's an advanced feature in the end, the user should know what he/she is doing,
2- I don't know what you mean here,
3- I definitely want to see that in the logs so it should be there,
4- I sometimes check the generated files for reference. Searching through the project folders is not practical for that.

I think we can use the same approach for 'namefile.qrc -> 'qrc_namefile.cpp': nevertheless we must each file manually intervene.
Sure, as I've mentioned before those enhancements are for general use, not just for Qt specific files (.ui, .qrc, ...) but for all custom tools (yacc, bison ...). If you mean the setting of custom extensions in advanced compiler options by 'manual intervention', theoretically the user will just have to do that only once in his/her entire Qt career. I don't think it's that bad.

Remains the most complicated, files containing macros to manage signals and slots.
I also have an enhancement of functionality for that. It will need a little user intervention though, but no more than a few clicks. And again it won't be specific to Qt. Yet a couple of bugs in the project loader and file addition code needs to be addressed first. I'm still waiting for developer opinions to continue as I'd like to move on step by step.

Faced with this difficulty, I used the last two years a plugin script that does all the work.
I have proposed a binary version. See the discussion http://forums.codeblocks.org/index.php/topic,20000.msg136816.html#msg136816
You could try it and give me your results.
I'm aware of your plugin and I think it's nice to have everything automated. But I don't think a plugin is the way to go for Qt. Reasons being:
1- Your plugin will be dead the moment you no longer maintain it and I don't think it will ever make its way into the officially supported plugins. Even if it makes it, it will still be dead when you quit maintaining it. There have been plugins before and they were all dead the moment sdk version changed because of lack of maintenance. Unfortunately that's how things go in CB (no offense meant to CB developers, it's just the way I see it is).
2- It will be specific to Qt usage, not general like for other custom tools.
3- I believe CB already has a capable build system to manage Qt and other tools which only needs simple enhancements.
I don't mean to discredit your plugin. I'm just expressing my reasons and I'll try it when I find some spare time.

Offline LETARTARE

  • Lives here!
  • ****
  • Posts: 531
  • L'ami de l'homme.The friend of man.
    • LETARTARE
Re: Enhancements to the CB Build System
« Reply #5 on: February 26, 2015, 01:10:24 pm »
@scarphin
thank you to answer me
Quote
I don't know what you mean here,
I meant: whenever one uses 'Build' everything is reconstructed.

Quote
the user will just have to do that only once in his/her entire Qt career
I work on two projects with many 'moc_namefile.cpp' (40 and  156), it may take time.

Your overall vision will actually be sustainable. Otherwise, your reasons are good.

If you have some time, I can get advice from a specialist Qt.

Regards

CB-13483, plugins-sdk-2.25.0 : Collector-2.0.0, AddOnForQt-3.9.1
1-Win7 Business Pack1 64bits : wx-3.2.4, gcc-8.1.0,
2-OpenSuse::Leap-15.4-64bits : wx-3.2.4;gtk3, gcc-8.2.1,
=> !! The messages are translated by Deepl

Offline scarphin

  • Lives here!
  • ****
  • Posts: 644
Re: Enhancements to the CB Build System
« Reply #6 on: February 26, 2015, 04:11:49 pm »
I meant: whenever one uses 'Build' everything is reconstructed.
Ok, I understand now. You're right and I think that results from CB not checking build dates/times for files with custom extensions to decide whether they must be rebuilt or not. It is in my todo list to check if CB can be made aware of the build dates/times of files with custom extensions so it doesn't build them when not needed. It has a low priority though.

Quote
the user will just have to do that only once in his/her entire Qt career
I work on two projects with many 'moc_namefile.cpp' (40 and  156), it may take time.
I think we are talking about different things here. Let me be more clear. For '.ui' files (or other files with custom extensions) after the settings in advanced compiler options is set for that extension, the user doesn't have to intervene with any other setting in the build process (except maybe for changes or upgrades) no matter how many Qt projects or '.ui' files he/she is working with. Obviously header files requiring 'moc' are different. As they have the standard '.h' extension, an '.h' file extension cannot be set in advanced compiler options. The 'custom build' dialogue in file properties can be used to assign a specific build command (i.e. moc.exe xxx.h -o moc_xxx.cpp) to each of them and the resulting 'moc_xxx.cpp' file needs to be added to the project but that is not practical. My intention is to provide a 'generated files' input to that dialogue and make the assignment of specific commands easier a bit. The user will have to set all 'Q_OBJECT' files manually but with very little effort to be exact. Again that is a further step and I don't want to advance without discussing this with the developers first.

However, a work-around for 'moc' I used in the past for private projects was to use a different extension (my choice was '.hxx' to profit from other features CB provides) for headers including 'Q_OBJECT'. Then I could set a custom extension for '.hxx' and everything was automated. I even modified the 'class wizard' to allow for a Qt setting and make the necessary adjustments automatically. I can provide a patch for it (class wizard) if you want to try. This approach isn't suitable for public projects or projects of other people though as the project itself needs to go under heavy changes to fit into the custom extensions system.

If you have some time, I can get advice from a specialist Qt.
Well, I'm not a Qt specialist nor a professional programmer. I may be considered as an intermediate Qt user but if you need my help or advice, I'll try to provide as much as I can. ;)

Offline scarphin

  • Lives here!
  • ****
  • Posts: 644
Re: Enhancements to the CB Build System
« Reply #7 on: March 04, 2015, 02:01:19 am »
I submitted 2 patches to fix 2 bugs:

3- When 2 different compilers have different generated files for the same extension and if they are used for different targets in the same project, generated files are assigned to all targets regardless of which target's compiler generates them. They should be only assigned to targets whose compilers generate them. Submitted patch associates a compiler ID array to each generated file to fix the issue. Patch here -> https://sourceforge.net/p/codeblocks/tickets/146/

4- When files are added to the project, not all generated files for all target's compilers are added to the project. The fix is very simple. Patch here -> https://sourceforge.net/p/codeblocks/tickets/147/

Please comment.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Enhancements to the CB Build System
« Reply #8 on: March 04, 2015, 02:06:39 am »
I've done it already in the tickets.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline scarphin

  • Lives here!
  • ****
  • Posts: 644
Re: Enhancements to the CB Build System
« Reply #9 on: March 05, 2015, 02:56:29 am »
I've provided the steps to produce in the tickets, I couldn't reply here before because the forum was spitting database errors. And please comment my previous patches too not just the bugs. ;)

Offline scarphin

  • Lives here!
  • ****
  • Posts: 644
Re: Enhancements to the CB Build System
« Reply #10 on: April 10, 2015, 08:47:57 pm »
Considering the amount of interest, I decided to implement all the features I had in mind without going step by step. I attached the patch for qt users to try it if they wish to. This patch is NOT for integration into cb (as it is a huge patch implementing everything) but for the adventurer to customize his/her cb installation. I've been testing the implemented features for about a month now and it seems stable on win7 x64. One last reminder, sometimes the modified project needs to be reloaded when build files are modified or a new target is created.

The patch is created against rev10201 and implements:
1- Default weight for files with custom extensions.
2- Implemented build files for custom builds in file properties dialogue.
3- Practical usage of settings from a predefined custom extension in file properties dialogue (intended for header files with Q_OBJECT).
4- Possibility to use $(project_dir) type of macros for generated files in advanced compiler options or build files in file properties dialogue.
5- Auto-creation of folders of generated or build files if they are not existent.
6- Implemented time-stamp checks for files with custom extensions so the target won't be rebuild if they are up-to-date.
7- Fixes a couple of bugs in the build system.

Example usage for qt projects:
1- Create the settings for .ui, .qrc, .moc (to be used for Q_OBJECT headers) extensions in advanced compiler options.
2- Create a qt project.
3- Add files.
4- Set the header files containing the Q_OBJECT macro to use the settings from the .moc extension in file properties dialogue.
5- Compile the project.

That will be all from me, I hope it will serve qt users' needs.

Edit: Corrected incorrect number order.
« Last Edit: April 13, 2015, 12:48:33 pm by scarphin »

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: Enhancements to the CB Build System
« Reply #11 on: April 11, 2015, 03:54:37 pm »
The patch is created against rev10201 and implements:
1- Auto-weight for files with custom extensions.
2- Implemented build files for custom builds in file properties dialogue.
3- Practical usage of settings from a predefined custom extension in file properties dialogue (intended for header files with Q_OBJECT).
3- Possibility to use $(project_dir) type of macros for generated files in advanced compiler options or build files in file properties dialogue.
4- Auto-creation of folders of generated or build files if they are not existent.
5- Implemented time-stamp checks for files with custom extensions so the target won't be rebuild if they are up-to-date.
6- Fixes a couple of bugs in the build system.
Thanks for the contribution, but it would be nice not to have all in one go. This makes testing way easier and also the acceptance of (parts of) the patch.

Also: Bullet 1) sounds scary to me: Everything that is "automised" should be off by default and have an option to enable it explicitly. Changing project file weights "automagically" can seriously break the build system for people that use those weights already. What is the policy for these "Auto-XXX" features in your patch?
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline scarphin

  • Lives here!
  • ****
  • Posts: 644
Re: Enhancements to the CB Build System
« Reply #12 on: April 11, 2015, 04:33:31 pm »
Well, I've already tried step-by-step and as you can see, your comment is the only worthy developer comment I got on this whole topic. I apologize if I missed anything. Please let me know if there is anything more I can do.

About bullet 1, sorry I wasn't clear on that. Let me explain it further. The auto-weight feature does NOT change the weight of any file (source, resource, custom extension or other...) already belongs to a project. It only effects the newly added files. For example, a '.ui' file in qt (similar to .xrc files in wxwidgets) must be compiled to a header file (ordinary '.h' file) to be included in other source files so before the project gets to get build, the '.ui' files must be compiled first all the time. Auto-weight automatizes this by setting a default weight to '.ui' files (say 40) in advanced compiler options and then every '.ui' file will have this weight (40) the moment they are added to the project helping the user to avoid setting each '.ui' files weight manually. To be more clear, assume the user has a '.ui' file in his/her project with a weight of say (70) for some reason, that file's weight will NOT be changed to (40) when a new custom extension for '.ui' is added in advanced compiler options. Again it is intended only for new files to be added to the project. Hope that clears the matter.

Thanks for your input.

Edit: On a second thought, I think it's better to rename auto-weight to default weight to avoid confusion. Done in the original post.
« Last Edit: April 11, 2015, 05:58:15 pm by scarphin »

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: Enhancements to the CB Build System
« Reply #13 on: April 11, 2015, 10:18:22 pm »
Could you provide a demo project or even better a tiny tutorial what to do to make a proper (tiny) qt project utilising this features?
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline scarphin

  • Lives here!
  • ****
  • Posts: 644
Re: Enhancements to the CB Build System
« Reply #14 on: April 13, 2015, 02:27:50 pm »
I attached a simple project which is an exact port of the qt example below to cb build system:
http://doc.qt.io/qt-4.8/qt-designer-calculatorform-example.html
Obviously a prebuilt qt4 library is needed to compile the project. To make things easier please create a 'qt4' entry in 'settings->global variables' with the following settings:
base: path_to_qt_library
include: $(#qt4)\include
lib: $(#qt4)\lib
bin: $(#qt4)\bin

Before trying the project please make sure the following entries in 'settings->compiler->other settings->advanced options->commands' are in place for the project's compiler:

1- Create a new extension named 'ui' with:
command line: $(#qt4.bin)\uic.exe $file -o $(project_dir)tmp\ui_$file_name.h
generated files: $(project_dir)tmp\ui_$file_name.h
weight: 30

2- Create a new extension named 'moc' with:
command line: $(#qt4.bin)\moc.exe $file -o $(project_dir)tmp\moc_$file_name.cpp
generated files: $(project_dir)tmp\moc_$file_name.cpp
weight: 40

Note: There is no need for a file with an 'moc' extension in qt. The 'moc' entry will be used to facilitate the configuration of files including a 'Q_OBJECT' macro and its purpose will be clear later.

I marked the new features with '(x)' in the demonstration below with their numbers in my post here:
http://forums.codeblocks.org/index.php/topic,20043.msg137653.html#msg137653

1- Load the project, there should be 3 files; 'main.cpp', 'calculatorform.cpp', 'calculatorform.h'
2- Add the file 'calculatorform.ui' in the project's folder to the project, when added it's weight should be '30' (1)
3- File 'calculatorform.h' contains a 'Q_OBJECT' macro so it must be compiled with qt's moc compiler. Make sure the project's compiler is selected during this step.
    a- Select file properties by right-clicking on file 'calculatorform.h' in 'management->projects' and clicking 'properties'.
    b- Go to 'advanced' tab.
    c- Check 'use custom build'.
    d- Check 'use settings from extension'. (3)
    e- Select 'moc' from the drop-down box. (3) The 'custom build command' and 'generated files' (2) settings from the 'moc' entry in advanced compiler options will be set for the file. Hence the 'moc' entry in advanced compiler options.
    f- Click 'ok'.
4- Now as everything is set, hit 'compile'. After the project is compiled, check the project's folder to find the auto-generated files 'ui_calculatorform.h' and 'moc_calculatorform.cpp' under the 'tmp' folder (4). Even if 'tmp' folder doesn't exist, it will be created during the build process (5).
5- Clicking 'build' again won't build the project again unless any of the source files (source or custom extension) is changed. (6)

Some notes:
1- Build files (generated files box in 'file properties->advanced' tab) may sometimes require the project to be reloaded if they are modified. That's because of the current implementation of file properties.  In the current implementation, even if the user chooses to cancel the changes, they are saved to the original file the moment the compiler is switched when 'custom build command' is not empty. I believe a proper implementation would require a copy constructor for 'ProjectFile' class (or some copy function etc...) to make a temporary copy to work on. I'm not experienced enough with the contents of 'ProjectFile' to implement one, sorry.
2- There is a bug occuring sometimes giving the error 'moc_somefile.o not found'. Cleaning and reloading the project works but I wasn't able to reproduce it. So if anyone can reproduce, please post the steps and I'll try to investigate.
3- Please ignore my previous patches on this thread as they do not cover some of the bugs correctly.
4- As mentioned many times previously, none of these features are qt specific. They can also be used with other tools in general.
5- My tests are only on windows. Although much of the code is platform independent there are some parts that needs to be tested under other suported OSes.

Thanks.