Author Topic: command line lenght limit on Windows  (Read 2275 times)

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
command line lenght limit on Windows
« on: April 24, 2020, 05:54:57 am »
Hi guys, I have a new problem,

I compile an mbed os project, and always get an error when linking.

When I tried to find the cause of the error, I found that the tail of the link instruction was missing.

So I copied the error message and found that its length is just 65535,
so I guess the maximum length of this instruction buffer is 64KB.

However, there are tens of thousands of connection object files in mbed os,
so the length of the link instruction may be far more than 64KB.

Can this problem be solved?



I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline stahta01

  • Lives here!
  • ****
  • Posts: 7019
    • My Best Post
Re: command line lenght limit on Windows
« Reply #1 on: April 24, 2020, 06:07:09 am »
One way to get around the line length limit is to use static libs.

So, group a number of files together and create an CB target to build a static library.
After doing this many times with all the files; you create another target that links several or all of the targets together.

Another was is to see if the Compiler being used support response files.

Tim S.
C Programmer working to learn more about C++ and Git.
On Windows 7 64 bit and Windows 10 32 bit.
On Debian Stretch, compiling CB Trunk against wxWidgets 3.0.
--
When in doubt, read the CB WiKi FAQ. http://wiki.codeblocks.org

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #2 on: April 24, 2020, 06:24:03 am »
One way to get around the line length limit is to use static libs.

So, group a number of files together and create an CB target to build a static library.
After doing this many times with all the files; you create another target that links several or all of the targets together.

Another was is to see if the Compiler being used support response files.

Tim S.

Thank you for your reply,

This CBP project is automatically generated by the official tool provided by Mbed OS, and manual modification will be very difficult.

And according to the type of MCU, there are many changes in the compilation conditions, so it may not be easy to use a static library.

If CB can display the link instruction completely in the information window instead of truncating it, then I can copy and save it as a txt file, and then call it in the command window to solve it.
I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline stahta01

  • Lives here!
  • ****
  • Posts: 7019
    • My Best Post
Re: command line lenght limit on Windows
« Reply #3 on: April 24, 2020, 06:33:55 am »
The command window is the source of the problem.

Tim S.
C Programmer working to learn more about C++ and Git.
On Windows 7 64 bit and Windows 10 32 bit.
On Debian Stretch, compiling CB Trunk against wxWidgets 3.0.
--
When in doubt, read the CB WiKi FAQ. http://wiki.codeblocks.org

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #4 on: April 24, 2020, 06:34:48 am »
Add content

This length limit is not a limitation of the Windows command line, but should be the maximum length limit of the CB instruction buffer.

Can the instruction buffer of CB be changed to solve this problem?
I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2850
Re: command line lenght limit on Windows
« Reply #5 on: April 24, 2020, 08:17:50 am »
You can try to enable the html build log and look there, if the problem is the wxWidhgets control, than this should resolve the problem...
I search currently but have not found any hard coded limit...

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12913
    • Travis build status
Re: command line lenght limit on Windows
« Reply #6 on: April 24, 2020, 09:22:24 am »
There is a limit at the OS level. This is solved by using response files, but we don't support this.

https://devblogs.microsoft.com/oldnewthing/20031210-00/?p=41553
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #7 on: April 24, 2020, 10:02:59 am »
You can try to enable the html build log and look there, if the problem is the wxWidhgets control, than this should resolve the problem...
I search currently but have not found any hard coded limit...

Thank you,

I tried it, but it didn't output the HTML log file. Does it have to be compiled successfully to have this file?

The CB is "Nightly builds" : The 18 April 2020 build (12064) is out.

My system is Windows-10 1909, 64bit.

I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #8 on: April 24, 2020, 10:07:30 am »
There is a limit at the OS level. This is solved by using response files, but we don't support this.

https://devblogs.microsoft.com/oldnewthing/20031210-00/?p=41553

OK, Should be the reason ...

Is there a way for CB to display the complete link command line in the log window instead of being truncated?

Or save it as a txt file?
I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2850
Re: command line lenght limit on Windows
« Reply #9 on: April 24, 2020, 01:49:57 pm »
It is also in the html log output file.
If this option is enabled (as it is in your case) the html output file should be in the same folder as the project file and have the same name as the project.
As far as i can tell not the command line is the problem, but the log output or not?
Because the compilation process is started, but the error message for the linker is not completely present in the error output?
Am i right, or do i miss understand something?

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #10 on: April 24, 2020, 05:04:46 pm »
It is also in the html log output file.
If this option is enabled (as it is in your case) the html output file should be in the same folder as the project file and have the same name as the project.
As far as i can tell not the command line is the problem, but the log output or not?
Because the compilation process is started, but the error message for the linker is not completely present in the error output?
Am i right, or do i miss understand something?

Hi,

I don't know what causes the HTML log file is not generated.

This link is done through g ++, so you see a red g ++ error, the error message length is 65535, it is obvious that you have lost the file that must be used by the link.

I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #11 on: April 24, 2020, 05:07:35 pm »
It is also in the html log output file.
If this option is enabled (as it is in your case) the html output file should be in the same folder as the project file and have the same name as the project.
As far as i can tell not the command line is the problem, but the log output or not?
Because the compilation process is started, but the error message for the linker is not completely present in the error output?
Am i right, or do i miss understand something?

This is the end of the error message.

... ... \TARGET_GD32F4XX\TARGET_GD32F450ZI\Periphe'      ---- >   There should be many files that need to be linked behind it, but it is missing.

« Last Edit: April 24, 2020, 05:12:07 pm by Chun Jiu »
I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline sodev

  • Regular
  • ***
  • Posts: 407
Re: command line lenght limit on Windows
« Reply #12 on: April 24, 2020, 05:34:00 pm »
Not sure if it is related, but the directory of your sources contain non-ascii characters, this might cause problems.

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #13 on: April 24, 2020, 06:00:30 pm »
Not sure if it is related, but the directory of your sources contain non-ascii characters, this might cause problems.

Hi,

It is normal ASCII characters on another computer in my office. Now this screenshot is just to reproduce the problem. It is my home computer.
I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2850
Re: command line lenght limit on Windows
« Reply #14 on: April 24, 2020, 09:12:29 pm »
Now, that i think about it. I somehow remember to have fixed a bug like this some time ago...
What version of codeblocks are you using? Have you tried the latest nightly?

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #15 on: April 25, 2020, 02:53:03 am »
Now, that i think about it. I somehow remember to have fixed a bug like this some time ago...
What version of codeblocks are you using? Have you tried the latest nightly?

Thank you,

This is a problem left over from the history of "Millennium Bug". The command line buffering of Windows starts from 2048, to 8192, 32768 ... Maybe it is now 65536.

Microsoft's suggestion is to pass the contents of the command line to it in a txt file, if it is indeed the problem.

https://support.microsoft.com/en-us/help/830473/command-prompt-cmd-exe-command-line-string-limitation

My codeblocks is the latest nightly, and its version number is  'build 12064'. Wondows 10 is 1909, 64bit.

Now there are two problems that need to be solved:

1. Previous command line length limitation.
2. The expected html log file does not appear. 

===================
How to reproduce this problem
===================

The Mbed OS is opensource's RTOS.

Note that this Mbed OS is about 1GB, and with the compiled object file, it may exceed 2GB.

1. Need to install python 2, pip, mbed-cli and its dependencies, and gnu gcc for arm.

    GNU Tools for Arm Embedded Processors 9-2019q4-major Release   
    https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm

2. Run cmd.exe to command line window, type command: pip install -U mbed-cli.
3. Go to the folder used for testing, type command: mbed new test-mbedos, it will download 1GB of content.
4. Go to the test-mbedos folder,  it is newly created by the mbed command.
5. Add a main.cpp file:

    #include "mbed.h"

    DigitalOut led1(LED1);

    #define SLEEP_TIME                  300 // (msec)

    int main()
    {
        int count = 0;
        while (true) {
            // Blink LED and wait 0.5 seconds
            led1 = !led1;
            wait_ms(SLEEP_TIME);
        }
    }

6. Then continue to enter the order: mbed export -i codeblocks -m  GD32_F450ZI

    D:\test\test-mbedos>mbed export -i codeblocks -m  GD32_F450ZI
    [mbed] Working path "D:\test\test-mbedos" (program)
    Scan: test-mbedos
   
    D:\test\test-mbedos>

7. If the installation of dependent packages(mbed-cli) fails, you need to delete their version numbers(in mbed-os/requirements.txt).
8. At this time, a CB project file was created (test-mbedos.cbp).
9. Set the path of gcc arm toolchain in CB to start compiling.
I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2850
Re: command line lenght limit on Windows
« Reply #16 on: April 25, 2020, 01:50:25 pm »
Hi, thank your for the detailed description.

Quote
1. Previous command line length limitation.
2. The expected html log file does not appear. 
I can reproduce nr 1 with a project i have created with the help of the following script on windows 7:

Code: [Select]
activeProject <- GetProjectManager().GetActiveProject();

filename <- _T("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.cpp");

for(local i = 0; i < 500; i+=1)
{
    local thisFileName = _T(i.tostring()) + _T("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.cpp");
    local path =  activeProject.GetCommonTopLevelPath() + _T("src") + wxFILE_SEP_PATH + thisFileName;
    IO.WriteFileContents(path , _T("void func") + _T(i.tostring()) + _T("() {}") );
    print("create file: " + path)

    GetProjectManager().AddFileToProject(path, activeProject, 0);

    //activeProject.AddFile(activeProject.GetActiveBuildTarget(), path, true, true, 1);

}

activeProject.Save();


The linking breaks the maximal command line length.
I am working on a solution and will post a patch. This probably will take some discussion, because it modifies the heart of c::b...

The second problem, i can not reproduce at the moment... I will first try to fix bug 1 and then i will look into 2)


Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #17 on: April 25, 2020, 06:28:55 pm »
Quote
I am working on a solution and will post a patch. This probably will take some discussion, because it modifies the heart of c::b...

The second problem, i can not reproduce at the moment... I will first try to fix bug 1 and then i will look into 2)

Thank you very much, looking forward to your good news :-)
I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2850
Re: command line lenght limit on Windows
« Reply #18 on: April 26, 2020, 11:22:05 am »
Here is a link to a patch, that adds the response files to codeblocks

https://sourceforge.net/p/codeblocks/tickets/953/



Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #19 on: April 27, 2020, 07:35:56 am »
Here is a link to a patch, that adds the response files to codeblocks

https://sourceforge.net/p/codeblocks/tickets/953/

Thank you,

It seems that this part of the work is more complicated, I wait for the Nightly builds with this function to test.
I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.

Offline Chun Jiu

  • Multiple posting newcomer
  • *
  • Posts: 68
  • My Girlfriend
    • EasilyGCC
Re: command line lenght limit on Windows
« Reply #20 on: May 01, 2020, 01:51:03 am »
Here is a link to a patch, that adds the response files to codeblocks

https://sourceforge.net/p/codeblocks/tickets/953/

Hi ,

I found a temporary solution, but it is not very convenient, especially for ordinary CB users.

In the settings of the compile , I replaced the original link command line with a script.

However, this solution is more dangerous. Ordinary users may cause the CB to crash unexpectedly but do not know how to repair it.

So I still look forward to the official solution, thank you.





Code: [Select]
$linker $libdirs -o $exe_output $link_objects $link_resobjects $link_options $libs
Replace

Code: [Select]
[[ local a = _T(@"$link_objects"); a.Replace(_T("\\"), _T("/"), true); IO.WriteFileContents(_T("objs.txt") , a); ]]
$linker $libdirs -o $exe_output @objs.txt $link_resobjects $link_options $libs
arm-none-eabi-objcopy.exe -O binary $(TARGET_OUTPUT_FILE) $(TARGET_OUTPUT_DIR)$(TARGET_OUTPUT_BASENAME).bin
« Last Edit: May 01, 2020, 02:48:13 am by Chun Jiu »
I love my girlfriend like c++!    :-)

http://pan.baidu.com/s/1feNwU
easilygcc is a gmail's email.