Author Topic: slooooow linking  (Read 9363 times)

Offline wilcomir

  • Single posting newcomer
  • *
  • Posts: 8
slooooow linking
« on: October 24, 2009, 03:19:58 pm »
Hi everybody!

I've installed codeblocks on my desktop pc, and it works well, but on my laptop it's tooo slow: I mean, compiling is of course very very fast, but linkin 10 lines (yes, it's 10) takes my pc 1:20, 80 seconds. I really can't understand the motivation, I've searched around and I've found something about the "number of processes for parallel building" option, located in "compiler and debugger settings" "other settings". I've set it to 3, as my laptop is a dual core, with no effect.
my laptop's got an intel core 2 DUO and 3Gb of ram, i use windows xp pro sp3, and the GNU compiler. all the codeblocks settings are the default ones, except the one I mentioned. I've also already tried to uninstall and install codeblocks, but it's still the same.

thank to everyone who will answer me!

Ciao
Vladimir

PS sorry about my english, but I'm Italian :D

Offline koso

  • Multiple posting newcomer
  • *
  • Posts: 58
Re: slooooow linking
« Reply #1 on: October 24, 2009, 04:12:56 pm »
Have you tried to link it without C::B => usign direct compiler/linker commands?

Setting more threads with "number of processes for parallel building" wont help with linking, becase linking one target is done only by one command. Number od processes will affect only compilation of more targets.

And btw. when the executable is linked, how large is it (filesize)?

Offline wilcomir

  • Single posting newcomer
  • *
  • Posts: 8
Re: slooooow linking
« Reply #2 on: October 25, 2009, 12:44:39 pm »
no, I didn't try to compile it by command line... I don't even know how to :D however, te exe size is about 500kb... I think it is toooo much :D maybe it's something about static/dinamic linking?

thanks!

zabzonk

  • Guest
Re: slooooow linking
« Reply #3 on: October 25, 2009, 02:04:21 pm »
It's certainly not a hardware issue - I do my C++ development on an ancient Pentium III with 256Mb RAM, with no problems. A common cause of slowdowns like this is your anti-virus software scanning things it needn't - have you tried linking with your AV software disabled?

Offline wilcomir

  • Single posting newcomer
  • *
  • Posts: 8
Re: slooooow linking
« Reply #4 on: October 25, 2009, 02:49:29 pm »
hi zabzonk! thanks for your tip, I'll try to disable my AV as soon as I'll be back to the university (I'm at home for the WE). however, I haven't experienced any other slowing...

thanks for helping!

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: slooooow linking
« Reply #5 on: October 26, 2009, 10:53:10 am »
have you tried linking with your AV software disabled?
Antivirus software may indeed be the problem. In particular Kaspersky and Norten will make your computer practically unusable for development, they're as bad or worse than the malware that they presumably prevent.

I've been using Kaspersky because you get indefinite free licenses for it from some big computer magazine here, and surely something that's free can't be so bad that it doesn't warrant the price, can it? This assumption is wrong. Kaspersky will increase compile times by about 3/4 and will render quad-core CPUs as well as low-latency storage such as a SSD RAID entirely useless. On top of that, it will perform some kind of "deep scan" on new executables that run immediately after being created (such as 100% of everything that comes out the IDE when you develop, duh), which will freeze the computer for 15-20 seconds.

Linking can already be a daunting task if no security malware is breaking your computer, as the linker has to fetch a lot of random tiny fragments from many files. You may see noticeable speedups if you use a ramdisk or a SSD to host your library files.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

zabzonk

  • Guest
Re: slooooow linking
« Reply #6 on: October 26, 2009, 01:15:12 pm »
> Linking can already be a daunting task

Actually, my experience is that linking is almost always much faster than compilation and I try to organise my projects a set of libraries for this very reason. On my system, it takes approximately the same time to link a project of 45 object files, a couple of custom libraries and the standard libraries as it does to compile a couple of c++source files. Admittedly, my hardware is sclerotic, but I've seen the same thing on Solaris enterprise class servers.


Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: slooooow linking
« Reply #7 on: October 27, 2009, 09:40:45 am »
Funny how much figures differ.

My machine compiles about 5-10 non-trivial C++ source files per second, being mostly bound by process startup and file access. Linking can, depending on what's in the project, take anywhere from 1-2 seconds for a smallish project, and up to 45-60 seconds for something big.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

zabzonk

  • Guest
Re: slooooow linking
« Reply #8 on: October 27, 2009, 12:15:24 pm »
It seems to me that logically compiling should be slower than linking. Consider what a compiler has to do:

- open source file and shunt it through the preprocessor
- open all the #included header files - this will almost always be dozens and maybe hundreds
- parse the source from the files - this involves a lot of non-trivial pattern matching and database (symbol table) manipulation. If templates are used, their Turing-complete nature means that the compilation can take an arbitrary (or even infinite) amount of time.
- optimise the results of the parse
- write results to object file
- repeat for next source file

Compare this with the link process:

- open a  relatively small number of files (all the .o files and libraries) These are in binary format and so do not need any sophisticated parsing
- look up code by name - for the libraries at least, there are indexes to speed this up
- fix up the code - a simple linear process
- write code to single executable file

This is much less computationally and I/O intensive than compiling and so should be faster. Of course, not all linkers are as smart as we would like them to be - people seem to prefer to work on compilers, for some strange  reason :-), and linking cannot be as easily parallelised as the compilation process.


« Last Edit: October 27, 2009, 06:24:39 pm by zabzonk »

Offline wilcomir

  • Single posting newcomer
  • *
  • Posts: 8
Re: slooooow linking
« Reply #9 on: October 27, 2009, 12:43:37 pm »
hi everybody!
I've tried to disable my AV, but nothing happened. it always takes over a minute to link a file... any other ideas?

thank you!

zabzonk

  • Guest
Re: slooooow linking
« Reply #10 on: October 27, 2009, 01:12:08 pm »
As someone else suggested, try linking from the command line to exclude CB from the equation. To do this:

a) Make sure that the GCC bin directory is on your path
b) Copy the link command line that gets displayed in the CB Build Log to a Windows command line prompt running in your  CB project's root directory.

In fact, it might be a good idea to post the link command line here too, so we can take a look at it.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12006
    • Travis build status
Re: slooooow linking
« Reply #11 on: October 27, 2009, 01:21:47 pm »
hi everybody!
I've tried to disable my AV, but nothing happened. it always takes over a minute to link a file... any other ideas?

thank you!
Is you harddrive heavily fragmented?
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

zabzonk

  • Guest
Re: slooooow linking
« Reply #12 on: October 27, 2009, 01:24:54 pm »
> Is you harddrive heavily fragmented?

This shouldn't really make much difference - the linker does most of its work in memory. My ancient machine has a FAT32 file system I havent defragmented in years, but it has no linker speed problems.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9496
Re: slooooow linking
« Reply #13 on: October 27, 2009, 01:37:49 pm »
any other ideas?
Try "process explorer" from "SysInternals" (http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx) and track what exactly happens during linking. Notice that you'll need to setup a filter properly so you only see the relevant parts. Otherwise you'll be flooded with information. ;-)
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline Biplab

  • Developer
  • Lives here!
  • *****
  • Posts: 1874
    • Biplab's Blog
Re: slooooow linking
« Reply #14 on: October 27, 2009, 04:38:49 pm »
have you tried linking with your AV software disabled?
Antivirus software may indeed be the problem. In particular Kaspersky and Norten will make your computer practically unusable for development, they're as bad or worse than the malware that they presumably prevent.

I've been using Kaspersky because you get indefinite free licenses for it from some big computer magazine here, and surely something that's free can't be so bad that it doesn't warrant the price, can it? This assumption is wrong. Kaspersky will increase compile times by about 3/4 and will render quad-core CPUs as well as low-latency storage such as a SSD RAID entirely useless. On top of that, it will perform some kind of "deep scan" on new executables that run immediately after being created (such as 100% of everything that comes out the IDE when you develop, duh), which will freeze the computer for 15-20 seconds.

I also have a similar experience. I am currently using K*spersky antivirus suite. It does increase the build time as it wants to freeze the whole pc to see if a newly created executable is a virus or not. For example when u run configure script in MSYS or in CygWin, every conftest.exe is checked for virus and other malwares. End result - super slow compilation.

I faced even other problem with it. While running a configure script under MSYS, it wants me to confirm that every process launched by sh.exe or any child process is safe to run or not. End result - millions of clicks to let it know that I want to run the configure script. Putting all related process to safe list yields no result.

I had to give up and resort to cross-compilation. When I really need to compile anything under MSYS, I have to turn off internet connection and antivirus & then everything compiles fine. Moral of the story is that antivirus programs are the most common culprit for pc slowdown.
Be a part of the solution, not a part of the problem.

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1442
    • CenizaSOFT
Re: slooooow linking
« Reply #15 on: October 27, 2009, 08:06:15 pm »
The linking stage can take a fairly huge amount of time. It all depends on the options, libraries and code itself.

At work I am using Intel C++ to compile the release version of the project I am working on (Visual C++'s linker either crashes with some decent optimizations, or it produces wrong code that crashes at runtime). For the release version I have some aggressive options set, including /Qipo and generating both generic code and target optimized code. Just compiling the source code can take a few minutes, but the linking stage takes more than 20 minutes. It takes it time to load all needed information (eating more than 1 GB of RAM in the process), and then it takes it more time to produce the DLL. Add to that that it does it twice (I guess once for each target).

Linkers are not always as dumb as one may think, and that costs time.

Offline wilcomir

  • Single posting newcomer
  • *
  • Posts: 8
Re: slooooow linking
« Reply #16 on: October 29, 2009, 02:00:52 pm »
hi all!

here's the command line displayed while compiling and linking

mingw32-g++.exe    -c "C:\Documents and Settings\Vladi\Documenti\cpp\prova.cpp" -o "C:\Documents and Settings\Vladi\Documenti\cpp\prova.o"
mingw32-g++.exe  -o "C:\Documents and Settings\Vladi\Documenti\cpp\prova.exe" "C:\Documents and Settings\Vladi\Documenti\cpp\prova.o"  
Process terminated with status 0 (1 minutes, 19 seconds)
0 errors, 0 warnings

I do not think my hard drive is heavily fragmented, the laptop is new, less than a month, and the hard disk is practically empty. however I'm trying to link without cb, I'll let you know.

thanks!

EDIT I've tried linking from cmd, and it's as slow as with cb. moreover, it produced an error:

collect2: cannot find '1d'

that disappeared as I've added the minigw bin path to the ambient variable PATH

any other ideas?

thanks again!
« Last Edit: October 29, 2009, 02:21:11 pm by wilcomir »

Offline koso

  • Multiple posting newcomer
  • *
  • Posts: 58
Re: slooooow linking
« Reply #17 on: October 29, 2009, 02:58:18 pm »
Thats very strange, especialy when prova.o isn't extremly large.

Can you provide some minimal example / source code? And also compiler version .. "mingw32-g++ -v"

zabzonk

  • Guest
Re: slooooow linking
« Reply #18 on: October 29, 2009, 03:12:15 pm »
> I've tried linking from cmd, and it's as slow as with cb

So it looks like this is not a CB issue - I still suspect that it as an AV or similar issue with your specific laptop hardware and/or software. You may well be better off asking this question in a forum where you will get a wider audience than you will here - I suggest trying StackOverflow at  http://stackoverflow.com.

Offline wilcomir

  • Single posting newcomer
  • *
  • Posts: 8
Re: slooooow linking
« Reply #19 on: October 31, 2009, 01:00:06 pm »
hi all!
here's what's produced when I write mingw32-g++ -v:
Code: [Select]
C:\Documents and Settings\Vladi>mingw32-g++ -v
Reading specs from C:/Programmi/CodeBlocks/MinGW/bin/../lib/gcc/mingw32/3.4.5/sp
ecs
Configured with: ../gcc-3.4.5/configure --with-gcc --with-gnu-ld --with-gnu-as -
-host=mingw32 --target=mingw32 --prefix=/mingw --enable-threads --disable-nls --
enable-languages=c,c++,f77,ada,objc,java --disable-win32-registry --disable-shar
ed --enable-sjlj-exceptions --enable-libgcj --disable-java-awt --without-x --ena
ble-java-gc=boehm --disable-libgcj-debug --enable-interpreter --enable-hash-sync
hronization --enable-libstdcxx-debug
Thread model: win32
gcc version 3.4.5 (mingw-vista special)

and here's what I always compile & link:

Code: [Select]
#include <iostream>
using namespace std;

int main () {
    cout << "insert a number";
    int n;
    cin >> n;
    cout << 2*n;
}

the .o file is about 2kb, while the generated .exe is nearly 500kb

thanks!

Offline stahta01

  • Lives here!
  • ****
  • Posts: 6636
    • My Best Post
Re: slooooow linking
« Reply #20 on: October 31, 2009, 01:26:26 pm »
hi all!
here's what's produced when I write mingw32-g++ -v:
Code: [Select]
C:\Documents and Settings\Vladi>mingw32-g++ -v
Reading specs from C:/Programmi/CodeBlocks/MinGW/bin/../lib/gcc/mingw32/3.4.5/sp
ecs
Configured with: ../gcc-3.4.5/configure --with-gcc --with-gnu-ld --with-gnu-as -
-host=mingw32 --target=mingw32 --prefix=/mingw --enable-threads --disable-nls --
enable-languages=c,c++,f77,ada,objc,java --disable-win32-registry --disable-shar
ed --enable-sjlj-exceptions --enable-libgcj --disable-java-awt --without-x --ena
ble-java-gc=boehm --disable-libgcj-debug --enable-interpreter --enable-hash-sync
hronization --enable-libstdcxx-debug
Thread model: win32
gcc version 3.4.5 (mingw-vista special)

and here's what I always compile & link:

Code: [Select]
#include <iostream>
using namespace std;

int main () {
    cout << "insert a number";
    int n;
    cin >> n;
    cout << 2*n;
}

the .o file is about 2kb, while the generated .exe is nearly 500kb

thanks!

Debugging and library code increases size.

Tim S.
C Programmer working to learn more about C++ and Git.
On Windows 7 64 bit and Windows 10 32 bit.
On Debian Stretch, compiling CB Trunk against wxWidgets 3.0.
--
When in doubt, read the CB WiKi FAQ. http://wiki.codeblocks.org

Offline wilcomir

  • Single posting newcomer
  • *
  • Posts: 8
Re: slooooow linking
« Reply #21 on: November 03, 2009, 11:23:38 am »
Debugging and library code increases size.

Tim S.

So what do you suggest?

thanks!

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: slooooow linking
« Reply #22 on: November 03, 2009, 12:48:27 pm »
the .o file is about 2kb, while the generated .exe is nearly 500kb
According to your commandline, you are not generating any symbols and not linking to any external libraries. Thus, a program with a 2 kB object file should have minimum PE size, i.e. 4 kB. If it doesn't, then maybe you have some malware on your computer which adds that code (possibly several ones, since 500 kB is quite a lot).

To be sure it really isn't just debug symbols, though, rename the prova.exe to something that doesn't look like a program, for example prova.txt.
Then run strip prova.txt. If the program is now about 4kB, all is good, then the increased size was only due to debug symbols. If it's still around 500 kB and it was really created from a 2 kB object file with a commandline as described above, then you may want to ring the alarm bells.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9496
Re: slooooow linking
« Reply #23 on: November 03, 2009, 01:06:46 pm »
the .o file is about 2kb, while the generated .exe is nearly 500kb
Then run strip prova.txt. If the program is now about 4kB, all is good, then the increased size was only due to debug symbols. If it's still around 500 kB and it was really created from a 2 kB object file with a commandline as described above, then you may want to ring the alarm bells.
Same here. I am using TDM's GCC 4.4.1(2). The file is as described, not linked against any additional libs (at least that's my purpose).
-> After compilation / linking: ~ 980 kB
-> After strip: ~500 kB.
Same thing with GCC 3.4.5:
-> After compilation / linking: ~ 500 kB
-> After strip: ~280 kB.

However - looking at was the linker accesses I see:
- crtbegin.o / crtend.o
- crt2.o
- libstdc++.a
- libmingw32.a
- libgcc.a
- libmoldname.a
- libmingwex.a
- libmsvcrt.a
- libuser32.a
- libkernel32.a
...which are all quite big in size.
If I look at the symbols of the final stripped exe I see references to MSVCRT and others.

However - my computer is 100% virus free, the same applies in a VM without network connection.

So what...?! I have this feeling that this discussion will lead to nowhere.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: slooooow linking
« Reply #24 on: November 03, 2009, 07:16:01 pm »
Surprising. Most of these should add nothing or next to nothing in my opinion, unless you really enforce it by using ultra-fat components of libstdc++. user32, kernel32, or msvcrt for example will only add some thunks for functions that your program actually uses, that's a few bytes for a small program (8 bytes per thunk unless you force alignment to be more than that), and maybe a kilobyte or two top for pretty much every program. libstdc++ can certainly add a few hundred kilobytes, but only if you actually use the stuff (and then it's not surprising).

I've just looked what the unit test program to one of my containers compiles to, as that's pretty much the smallest non-helloworld project I have at hand. Gcc 4.4.0-tdm turns roughly 10 kB of sources (not counting system headers) into a 8.5 kB object file, and this into a 33 kB executable. That includes around 7 kB for the main program using std::vector, and 1 kB for exception handling. It takes well under a second to compile and link. Gcc 4.4.1-tdm produces considerably larger executables, though I don't have it installed now, so can't tell for this example.

Well, whichever it is, you're probably right insofar as we'll not likely be able to tell why wilcomir's executables are that big.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline wilcomir

  • Single posting newcomer
  • *
  • Posts: 8
Re: slooooow linking
« Reply #25 on: November 04, 2009, 02:52:24 pm »
According to your commandline, you are not generating any symbols and not linking to any external libraries. Thus, a program with a 2 kB object file should have minimum PE size, i.e. 4 kB. If it doesn't, then maybe you have some malware on your computer which adds that code (possibly several ones, since 500 kB is quite a lot).

To be sure it really isn't just debug symbols, though, rename the prova.exe to something that doesn't look like a program, for example prova.txt.
Then run strip prova.txt. If the program is now about 4kB, all is good, then the increased size was only due to debug symbols. If it's still around 500 kB and it was really created from a 2 kB object file with a commandline as described above, then you may want to ring the alarm bells.

hi, I did as you say, now the exe is 270kb in size.. still too much i believe... maybe I'll solve using another compiler, but wich one?

thanks everybody