New UserVarManager implementation

[BlueHazzard]

Hello,
since the  past view years i work on and off on the UserVariableManager... Now i am finally pleased to have a working patch against svn head...

From my point of view (correct me if i am wrong) global user variables should be used to configure external libraries (path, compiler and linker flags) that are different on each pc. On linux this is less used than on windows. To fulfil this functionality it should have the following properties:
1) Flexible naming (different sets, different variable names (ex. wx), different members of the variable (ex. include, library ecc.)
2) Definable over command line for batch builds
3) Inform the user if a variable is not defined and give some hints what the variable is used for

What it is NOT meant to do (at the moment)
*) Automatically detect a library or paths to a library

With the current implementation only 1) is implemented, 2) over a dirty hack and 3) partially and fails on headless batch builds. Batch building is tricky because when a variable is not defined no messages are logged and on headless servers the build simply was stuck....
With the attached patch 2 is possible and 3 partially (at least on informing the user), and with a follow up patch also 3 is better implemented.

What does this patch:
1) The attached patch does NOT modify the behaviour of global variables from a user perspective! Old projects should behave the same. If not it is a bug!
2) It adds the command line flag '-D variable[.member]=value' that can be repeated multiple times for different variables. If the variable is not stored in the current configuration it is temporary generated. This allows to set and override global variables via command line.
3) It adds the command line flag -S setname to activate a specific set (this would currently be possible with different personalities, but it is extremely cumbersome)
4) It stores the variables in memory and does not load them from configuration file every time they are used during compiling (should theoretically speed up compile times)
5) It adds an cancel/save button to the edit variable dialog, so the user can cancel a erroneous input
6) Adds a lot API and documentation to UserVariableManager so it can be used in other plugins or over scripting
6.1) Api for get and set variable sets, variables and members
6.2) Api to get and set the whole set of variables
6.3) Api to store the variables to configuration file
6.4) Api to parse variable names
6.5) Api to check if a variable exists in the first place
7) Moves all related dialogs from sdk to src
Introduces an UI interface so we can use different UIs for batch builds (console) and UI builds
9) More information is logged to the codeblocks log if something went wrong (this helps on headless servers A LOT)
10) Remove any config file writing that has happened in user variable dialogs to the new api of the UserVariableManager  so configuration file loading and storing are in one single place

I have looked over this patch 100 times, but i think i am now blind for any (format) errors...

I have not already incremented the SDK API number, because this api is not stable until after the next patch...

Any comments?

[omlk]

Hi, it would be great if you could make a video showing all the possibilities.

[BlueHazzard]

There are not really new features to see on a video (this is intended, no change the user behaviour), beside the command line arguments and not blocking on batch builds...

[AndrewCot]

Now I have come up for air I will have a look at it in the next day or two as I am heavily using the global variables for my local builds.

I just merged my Linux and Mac project files and created yet more global variables in order for the merge to work as I kept having to make the change twice due to supporting two sets of project files when 95+% was common between the two sets.

[AndrewCot]

First problem:

The patch file does not include the <Unit filename="src/resources/ask_global_uservar.xrc"> new that's in the project files and in the Github branch, but not in the patch.

[AndrewCot]

Is it expected that the patch includes the previous changes as per the following thread for the tabGlobVars dialog as I cannot see it in the patch? Add UI to store default values and descriptions of global variables in project

This is a query, how do I get to the UI in point 8 ()?
8. Introduces an UI interface so we can use different UIs for batch builds (console) and UI builds

I will need to spend a bit of time trying to use the code once all of the changes are sorted out.

[AndrewCot]

Second problem:
This could be caused by me missing something in the code merge, but I cannot add a new user variable (one of the existing 7 user definable on the right) . I cannot see where the m_Name or m_Value is updated with the values entered before the UsrGlblMgrEditDialog::SaveSet is called when you click on the saveSet button.

I added the following line in the UserVariableManager::Save() function after the write to save me debugging the structures as it was too slow for me:

Manager::Get()->GetLogManager()->DebugLog(wxString::Format("UserVariableManager::Save - manager->Write(%s, %s);", basePath + member, value));


UPDATE:Adding a call to "Save();" as the first line of UsrGlblMgrEditDialog::SaveSet looked like it fixed the save of the new user variable. I only did one test as I am calling it for today.

[AndrewCot]

I have one enhancement request as follows:
a) Add an extra user variable, so it takes the total to 8. A side effect is that the GUI has a better layout IMHO.
b) In src\src\uservardlgs.cpp change the references to the max user variables from 7 to MAX_USER_DEFINED (or another another variable )
    and add a "#define MAX_USER_DEFINED 8" at the top of the file after the #include's. This will make it easier if in the future the number needs increasing.
c) In the src\src\resources\global_uservars.xrc add the n7 and v7 for the new user variables

[cacb]

I use C::B global variables extensively, so I will be reading this thread with interest. Thanks for the good work.

[BlueHazzard]

Thank you for testing all!

8. Introduces an UI interface so we can use different UIs for batch builds (console) and UI builds

This is only meant as interface class in code (virtual class UserVarManagerUI that has two implementations: UserVarManagerGUI for GUI and UserVarManagerNoGuiUI for non GUI things, this at the moment does nothing, but output some in the log and does not block  )

The patch file does not include the <Unit filename="src/resources/ask_global_uservar.xrc"> new that's in the project files and in the Github branch, but not in the patch.

I will add this in a follow up patch. At the moment i wait a bit for Morten MacFly to give a green light for this patch, then i will follow up with the next.

I added the following line in the UserVariableManager::Save() function after the write to save me debugging the structures as it was too slow for me:

Manager::Get()->GetLogManager()->DebugLog(wxString::Format("UserVariableManager::Save - manager->Write(%s, %s);", basePath + member, value));


UPDATE:Adding a call to "Save();" as the first line of UsrGlblMgrEditDialog::SaveSet looked like it fixed the save of the new user variable. I only did one test as I am calling it for today.

Yes this is a error i will look into. This probably happened because i had to split up my git history and create a svn patch, this is all a bit dangerous...

I have one enhancement request as follows:
a) Add an extra user variable, so it takes the total to 8. A side effect is that the GUI has a better layout IMHO.
b) In src\src\uservardlgs.cpp change the references to the max user variables from 7 to MAX_USER_DEFINED (or another another variable )
    and add a "#define MAX_USER_DEFINED 8" at the top of the file after the #include's. This will make it easier if in the future the number needs increasing.
c) In the src\src\resources\global_uservars.xrc add the n7 and v7 for the new user variables

Thank you for finding this. My ultimate target is to get rid of any fixed number of member variables and rework this UI to use a wxDataViewCtrl  so a user can enter unlimited number of members
But same as above before i put more work into this i would like some kind of green light from Morten

[AndrewCot]

Do you have a branch somewhere with a enhanced/working version with the dlgAskGlobalUsrVar dialog wired up and working? I would like to test it with this in it as this the missing bit from the two patches. I have merged all of the changes and the updates to the previous patch from the test/global_user_var_to_prject/1 branch.

So far from an end user perspective it looks and feels like nothing has changed. Also from my look at the patch and merging the changes the code changes have impacted on very few files, which very good and I like it as this means that most devs will not notice any changes as no plugins had to change.

Overall it looks like a very good replacement for the existing code. I still need to do some more testing in the next week using it in the real world.

[AndrewCot]

Another bug:On Linux Ubuntu 22.04 with wxWidget 3.0.5 UsrGlblMgrEditDialog::UpdateChoices() crashes with the following when I try to open the global variables dialog when you specify a new config file with no global variables defined with the following bt:

Code

In  ()
#2  0x0000555555777c9a in std::__detail::_Map_base<wxString, std::pair<wxString const, std::unordered_map<wxString, UserVariable, std::hash<wxString>, std::equal_to<wxString>, std::allocator<std::pair<wxString const, UserVariable> > > >, std::allocator<std::pair<wxString const, std::unordered_map<wxString, UserVariable, std::hash<wxString>, std::equal_to<wxString>, std::allocator<std::pair<wxString const, UserVariable> > > > >, std::__detail::_Select1st, std::equal_to<wxString>, std::hash<wxString>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::at (this=0x7fffffffd610, __k=...) at /usr/include/c++/11/bits/hashtable_policy.h:758
/usr/include/c++/11/bits/hashtable_policy.h:758:23678:beg:0x555555777c9a
At /usr/include/c++/11/bits/hashtable_policy.h:758
#2  0x0000555555777c9a in std::__detail::_Map_base<wxString, std::pair<wxString const, std::unordered_map<wxString, UserVariable, std::hash<wxString>, std::equal_to<wxString>, std::allocator<std::pair<wxString const, UserVariable> > > >, std::allocator<std::pair<wxString const, std::unordered_map<wxString, UserVariable, std::hash<wxString>, std::equal_to<wxString>, std::allocator<std::pair<wxString const, UserVariable> > > > >, std::__detail::_Select1st, std::equal_to<wxString>, std::hash<wxString>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::at (this=0x7fffffffd610, __k=...) at /usr/include/c++/11/bits/hashtable_policy.h:758
/usr/include/c++/11/bits/hashtable_policy.h:758:23678:beg:0x555555777c9a
At /mnt/d/Andrew_Development/Work_Installers/CodeBLocks_Private_Experimental_GCC/src/src/uservardlgs.cpp:527
#4  0x000055555576d318 in UsrGlblMgrEditDialog::UpdateChoices (this=0x7fffffffd170) at /mnt/d/Andrew_Development/Work_Installers/CodeBLocks_Private_Experimental_GCC/src/src/uservardlgs.cpp:527
/mnt/d/Andrew_Development/Work_Installers/CodeBLocks_Private_Experimental_GCC/src/src/uservardlgs.cpp:527:15867:beg:0x55555576d318
At /mnt/d/Andrew_Development/Work_Installers/CodeBLocks_Private_Experimental_GCC/src/src/uservardlgs.cpp:527
#4  0x000055555576d318 in UsrGlblMgrEditDialog::UpdateChoices (this=0x7fffffffd170) at /mnt/d/Andrew_Development/Work_Installers/CodeBLocks_Private_Experimental_GCC/src/src/uservardlgs.cpp:527
/mnt/d/Andrew_Development/Work_Installers/CodeBLocks_Private_Experimental_GCC/src/src/uservardlgs.cpp:527:15867:beg:0x55555576d318

Line 527 is:

Code

 vars.reserve(m_varMap.at(m_CurrentSetName).size());

[BlueHazzard]

Thank you for reporting. This should be fixed in the branch "test/global_user_var_to_project/1/rebase/before_default_var/1" on my github repo
this branch is the branch before introducing the default variables in the project file.
the branch "test/global_user_var_to_project/1/rebase/after_default_var/1" introduces the default variables in the project

[AndrewCot]

Thanks for making the changes. I will grab the git update today or tomorrow and give it a go.

[AndrewCot]

The uservardlgs.cpp file does not compile on the Mac without changing the includes to the following:

Code

#include <wx/dirdlg.h>
#include <wx/filedlg.h>
#include <wx/sizer.h>
#include <wx/stattext.h>

#include "uservardlgs.h"
#include "annoyingdialog.h"
#include "tinywxuni.h"

[AndrewCot]

An issue I just came across and it's in the existing implementation a well is that there are no errors in the log if a variable is not found due to using the wrong variable set. May be an idea to add logging of errors/warnings when variables are not found.

[BlueHazzard]

My idea was to return a error log in the ReplaceMacros function. So it could be outputed to the compiler log during compiling and in the codeblocks log when other. I mean if i compile something and then it does not work, it would be nice to see in the compiler log...

[AndrewCot]

Sounds good.

[ollydbg]

My idea was to return a error log in the ReplaceMacros function. So it could be outputed to the compiler log during compiling and in the codeblocks log when other. I mean if i compile something and then it does not work, it would be nice to see in the compiler log...

I totally agree with this idea.

[AndrewCot]

I could not find the implementation of the following:
    wxString GetMemberValue(const wxString & setName, const wxString & varName, const wxString & memberName) const;