Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => Topic started by: Chun Jiu on April 24, 2020, 05:54:57 am

Title: command line lenght limit on Windows
Post by: Chun Jiu on April 24, 2020, 05:54:57 am
Hi guys, I have a new problem,

I compile an mbed os project, and always get an error when linking.

When I tried to find the cause of the error, I found that the tail of the link instruction was missing.

So I copied the error message and found that its length is just 65535,
so I guess the maximum length of this instruction buffer is 64KB.

However, there are tens of thousands of connection object files in mbed os,
so the length of the link instruction may be far more than 64KB.

Can this problem be solved?



Title: Re: command line lenght limit on Windows
Post by: stahta01 on April 24, 2020, 06:07:09 am
One way to get around the line length limit is to use static libs.

So, group a number of files together and create an CB target to build a static library.
After doing this many times with all the files; you create another target that links several or all of the targets together.

Another was is to see if the Compiler being used support response files.

Tim S.
Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 24, 2020, 06:24:03 am
One way to get around the line length limit is to use static libs.

So, group a number of files together and create an CB target to build a static library.
After doing this many times with all the files; you create another target that links several or all of the targets together.

Another was is to see if the Compiler being used support response files.

Tim S.

Thank you for your reply,

This CBP project is automatically generated by the official tool provided by Mbed OS, and manual modification will be very difficult.

And according to the type of MCU, there are many changes in the compilation conditions, so it may not be easy to use a static library.

If CB can display the link instruction completely in the information window instead of truncating it, then I can copy and save it as a txt file, and then call it in the command window to solve it.
Title: Re: command line lenght limit on Windows
Post by: stahta01 on April 24, 2020, 06:33:55 am
The command window is the source of the problem.

Tim S.
Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 24, 2020, 06:34:48 am
Add content

This length limit is not a limitation of the Windows command line, but should be the maximum length limit of the CB instruction buffer.

Can the instruction buffer of CB be changed to solve this problem?
Title: Re: command line lenght limit on Windows
Post by: BlueHazzard on April 24, 2020, 08:17:50 am
You can try to enable the html build log and look there, if the problem is the wxWidhgets control, than this should resolve the problem...
I search currently but have not found any hard coded limit...
Title: Re: command line lenght limit on Windows
Post by: oBFusCATed on April 24, 2020, 09:22:24 am
There is a limit at the OS level. This is solved by using response files, but we don't support this.

https://devblogs.microsoft.com/oldnewthing/20031210-00/?p=41553
Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 24, 2020, 10:02:59 am
You can try to enable the html build log and look there, if the problem is the wxWidhgets control, than this should resolve the problem...
I search currently but have not found any hard coded limit...

Thank you,

I tried it, but it didn't output the HTML log file. Does it have to be compiled successfully to have this file?

The CB is "Nightly builds" : The 18 April 2020 build (12064) is out.

My system is Windows-10 1909, 64bit.

Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 24, 2020, 10:07:30 am
There is a limit at the OS level. This is solved by using response files, but we don't support this.

https://devblogs.microsoft.com/oldnewthing/20031210-00/?p=41553

OK, Should be the reason ...

Is there a way for CB to display the complete link command line in the log window instead of being truncated?

Or save it as a txt file?
Title: Re: command line lenght limit on Windows
Post by: BlueHazzard on April 24, 2020, 01:49:57 pm
It is also in the html log output file.
If this option is enabled (as it is in your case) the html output file should be in the same folder as the project file and have the same name as the project.
As far as i can tell not the command line is the problem, but the log output or not?
Because the compilation process is started, but the error message for the linker is not completely present in the error output?
Am i right, or do i miss understand something?
Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 24, 2020, 05:04:46 pm
It is also in the html log output file.
If this option is enabled (as it is in your case) the html output file should be in the same folder as the project file and have the same name as the project.
As far as i can tell not the command line is the problem, but the log output or not?
Because the compilation process is started, but the error message for the linker is not completely present in the error output?
Am i right, or do i miss understand something?

Hi,

I don't know what causes the HTML log file is not generated.

This link is done through g ++, so you see a red g ++ error, the error message length is 65535, it is obvious that you have lost the file that must be used by the link.

Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 24, 2020, 05:07:35 pm
It is also in the html log output file.
If this option is enabled (as it is in your case) the html output file should be in the same folder as the project file and have the same name as the project.
As far as i can tell not the command line is the problem, but the log output or not?
Because the compilation process is started, but the error message for the linker is not completely present in the error output?
Am i right, or do i miss understand something?

This is the end of the error message.

... ... \TARGET_GD32F4XX\TARGET_GD32F450ZI\Periphe'      ---- >   There should be many files that need to be linked behind it, but it is missing.

Title: Re: command line lenght limit on Windows
Post by: sodev on April 24, 2020, 05:34:00 pm
Not sure if it is related, but the directory of your sources contain non-ascii characters, this might cause problems.
Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 24, 2020, 06:00:30 pm
Not sure if it is related, but the directory of your sources contain non-ascii characters, this might cause problems.

Hi,

It is normal ASCII characters on another computer in my office. Now this screenshot is just to reproduce the problem. It is my home computer.
Title: Re: command line lenght limit on Windows
Post by: BlueHazzard on April 24, 2020, 09:12:29 pm
Now, that i think about it. I somehow remember to have fixed a bug like this some time ago...
What version of codeblocks are you using? Have you tried the latest nightly?
Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 25, 2020, 02:53:03 am
Now, that i think about it. I somehow remember to have fixed a bug like this some time ago...
What version of codeblocks are you using? Have you tried the latest nightly?

Thank you,

This is a problem left over from the history of "Millennium Bug". The command line buffering of Windows starts from 2048, to 8192, 32768 ... Maybe it is now 65536.

Microsoft's suggestion is to pass the contents of the command line to it in a txt file, if it is indeed the problem.

https://support.microsoft.com/en-us/help/830473/command-prompt-cmd-exe-command-line-string-limitation (https://support.microsoft.com/en-us/help/830473/command-prompt-cmd-exe-command-line-string-limitation)

My codeblocks is the latest nightly, and its version number is  'build 12064'. Wondows 10 is 1909, 64bit.

Now there are two problems that need to be solved:

1. Previous command line length limitation.
2. The expected html log file does not appear. 

===================
How to reproduce this problem
===================

The Mbed OS is opensource's RTOS.

Note that this Mbed OS is about 1GB, and with the compiled object file, it may exceed 2GB.

1. Need to install python 2, pip, mbed-cli and its dependencies, and gnu gcc for arm.

    GNU Tools for Arm Embedded Processors 9-2019q4-major Release   
    https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm (https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm)

2. Run cmd.exe to command line window, type command: pip install -U mbed-cli.
3. Go to the folder used for testing, type command: mbed new test-mbedos, it will download 1GB of content.
4. Go to the test-mbedos folder,  it is newly created by the mbed command.
5. Add a main.cpp file:

    #include "mbed.h"

    DigitalOut led1(LED1);

    #define SLEEP_TIME                  300 // (msec)

    int main()
    {
        int count = 0;
        while (true) {
            // Blink LED and wait 0.5 seconds
            led1 = !led1;
            wait_ms(SLEEP_TIME);
        }
    }

6. Then continue to enter the order: mbed export -i codeblocks -m  GD32_F450ZI

    D:\test\test-mbedos>mbed export -i codeblocks -m  GD32_F450ZI
    [mbed] Working path "D:\test\test-mbedos" (program)
    Scan: test-mbedos
   
    D:\test\test-mbedos>

7. If the installation of dependent packages(mbed-cli) fails, you need to delete their version numbers(in mbed-os/requirements.txt).
8. At this time, a CB project file was created (test-mbedos.cbp).
9. Set the path of gcc arm toolchain in CB to start compiling.
Title: Re: command line lenght limit on Windows
Post by: BlueHazzard on April 25, 2020, 01:50:25 pm
Hi, thank your for the detailed description.

Quote
1. Previous command line length limitation.
2. The expected html log file does not appear. 
I can reproduce nr 1 with a project i have created with the help of the following script on windows 7:

Code
activeProject <- GetProjectManager().GetActiveProject();

filename <- _T("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.cpp");

for(local i = 0; i < 500; i+=1)
{
    local thisFileName = _T(i.tostring()) + _T("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.cpp");
    local path =  activeProject.GetCommonTopLevelPath() + _T("src") + wxFILE_SEP_PATH + thisFileName;
    IO.WriteFileContents(path , _T("void func") + _T(i.tostring()) + _T("() {}") );
    print("create file: " + path)

    GetProjectManager().AddFileToProject(path, activeProject, 0);

    //activeProject.AddFile(activeProject.GetActiveBuildTarget(), path, true, true, 1);

}

activeProject.Save();


The linking breaks the maximal command line length.
I am working on a solution and will post a patch. This probably will take some discussion, because it modifies the heart of c::b...

The second problem, i can not reproduce at the moment... I will first try to fix bug 1 and then i will look into 2)

Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 25, 2020, 06:28:55 pm
Quote
I am working on a solution and will post a patch. This probably will take some discussion, because it modifies the heart of c::b...

The second problem, i can not reproduce at the moment... I will first try to fix bug 1 and then i will look into 2)

Thank you very much, looking forward to your good news :-)
Title: Re: command line lenght limit on Windows
Post by: BlueHazzard on April 26, 2020, 11:22:05 am
Here is a link to a patch, that adds the response files to codeblocks

https://sourceforge.net/p/codeblocks/tickets/953/


Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 27, 2020, 07:35:56 am
Here is a link to a patch, that adds the response files to codeblocks

https://sourceforge.net/p/codeblocks/tickets/953/

Thank you,

It seems that this part of the work is more complicated, I wait for the Nightly builds with this function to test.
Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on May 01, 2020, 01:51:03 am
Here is a link to a patch, that adds the response files to codeblocks

https://sourceforge.net/p/codeblocks/tickets/953/

Hi ,

I found a temporary solution, but it is not very convenient, especially for ordinary CB users.

In the settings of the compile , I replaced the original link command line with a script.

However, this solution is more dangerous. Ordinary users may cause the CB to crash unexpectedly but do not know how to repair it.

So I still look forward to the official solution, thank you.

(http://2020-05-01 07_40_36-control.png)

(http://2020-05-01 07_41_22-main.cpp [mbed-git-lib] - Code__Blocks svn build)

Code
$linker $libdirs -o $exe_output $link_objects $link_resobjects $link_options $libs

Replace

Code
[[ local a = _T(@"$link_objects"); a.Replace(_T("\\"), _T("/"), true); IO.WriteFileContents(_T("objs.txt") , a); ]]
$linker $libdirs -o $exe_output @objs.txt $link_resobjects $link_options $libs
arm-none-eabi-objcopy.exe -O binary $(TARGET_OUTPUT_FILE) $(TARGET_OUTPUT_DIR)$(TARGET_OUTPUT_BASENAME).bin
Title: Re: command line lenght limit on Windows
Post by: Chun Jiu on April 22, 2021, 07:06:07 am
Here is a link to a patch, that adds the response files to codeblocks

https://sourceforge.net/p/codeblocks/tickets/953/

Hi BlueHazzard,

This is a late thank you, I confirm that your work has helped me solve the problem.

Those projects that could not be correctly compiled and linked before,

Now it can compile and link correctly after testing.

Thanks again!