Author Topic: Quick&dirty C++ language conformance tests  (Read 12945 times)

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Quick&dirty C++ language conformance tests
« on: December 16, 2005, 02:45:09 pm »
Hello,

I have found this website which provides quick&dirty C++ language conformance tests for the Win32 platform. As C::B supports them, this website could be useful for deciding which compiler is best suitable for a specific project.

Michael

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: Quick&dirty C++ language conformance tests
« Reply #1 on: December 16, 2005, 06:19:08 pm »
It appears then, that the most c++ standard compliant compiler is:
-GCC >=3.2 at the top
-DMars >=8.45 very good also
-MSVC >=7.1 rather good nonetheless

-OpenWatcom <=1.3 pretty bad
-OpenWatcom >=1.4 Dev catching up, but still worse than others

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Quick&dirty C++ language conformance tests
« Reply #2 on: December 16, 2005, 06:28:27 pm »
Despite all, GNU GCC seems to be a good choice (I think there was a thread about comparing speed, executable size, etc of several compiler. May be an Intel Compiler C++ thread).

MSVC is also improving. MSVC 2005 Express seems also to compile library as Loki (see http://msdn.microsoft.com/vstudio/express/visualc/features/language/default.aspx), which is not so evident.

But none is absolutly perfect...there is still some work to do :).

Michael

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Quick&dirty C++ language conformance tests
« Reply #3 on: December 16, 2005, 09:22:20 pm »
I think this comparison (although still in favour to gcc) is quite unfair. They compare a 3 year old version of gcc with the current release of DM and the current release of Microsoft Visual C++. However, it makes me smile that gcc still beats them.


On a different subject, I recently found out something that I'd like to share with the many people who keep bashing gcc for its poor executable size. Read the manual. Yes, it sounds like a stupid recommendation, but seriously... do it.

You can not only tell the compiler to optimize the code, but you can also pass optimization flags to the linker. This is something few people seem to know (I did not know myself until two weeks ago). By passing -O2 to the linker, I managed to reduce the executable size of a program of mine by 18%.  :)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: Quick&dirty C++ language conformance tests
« Reply #4 on: December 16, 2005, 09:38:26 pm »
GNU/GCC is not that bad regarding executable size (still worse compared to others).

When it looses and badly, is in compilation/linkage time. I mean, compiling wx+cb can take more than 30 minutes in my new pc with mingw 3.4.
However, I have hopes that GCC post-4.1 will be much faster thanks to the new optimizations and features it introduces.

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1442
    • CenizaSOFT
Re: Quick&dirty C++ language conformance tests
« Reply #5 on: December 17, 2005, 04:45:59 pm »
-O2 to the linker? 18%? I just tried that and the filesize is still the same.

-Wl,-O2

Am I missing something?

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Quick&dirty C++ language conformance tests
« Reply #6 on: December 17, 2005, 05:18:05 pm »
-O2 to the linker? 18%? I just tried that and the filesize is still the same.

-Wl,-O2

Am I missing something?
I don't know really, lol. When I read that in the manpages, I immediately tested it, and it turned a 802kB app into a 658kB app, which made me go "wowww!". I then tried recompiling wxWidgets in that manner, but it did not seem to make a difference (although I am not sure whether or not I did it correctly. You know compiling wxWidgets is not precisely intuitive...). And since building wx takes forever, I did not bother any further. So far my experience with it.
Personally, I have currently no need to squeeze every bit out of my applications, so I prefer the linker running faster instead :)
But still, there is more in it than the eye meets.

Although I don't know how the linker optimizes at all, I guess it is mostly useful on large executables because you have the minimum overhead of 4k due to sections (at least using PE, don't know the limits of ELF) which is less noticeable the bigger the executable is, and because most likely "optimize" means dead-stripping unused code and playing with function alignments where it does not break anything.

The most notable thing, however, is that nobody (including myself) ever reads the documentation (yes, reading documentation sucks almost as much as writing), but everybody thinks he knows the tools he is using.
And then one day after many years, you discover that you don't know anything at all... I have half an hour of reading documentations on my daily schedule now :)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Urxae

  • Regular
  • ***
  • Posts: 376
Re: Quick&dirty C++ language conformance tests
« Reply #7 on: December 17, 2005, 05:55:08 pm »
-O2 to the linker? 18%? I just tried that and the filesize is still the same.

-Wl,-O2

Am I missing something?
I don't know really, lol. When I read that in the manpages, I immediately tested it, and it turned a 802kB app into a 658kB app, which made me go "wowww!". I then tried recompiling wxWidgets in that manner, but it did not seem to make a difference (although I am not sure whether or not I did it correctly. You know compiling wxWidgets is not precisely intuitive...). And since building wx takes forever, I did not bother any further. So far my experience with it.
Personally, I have currently no need to squeeze every bit out of my applications, so I prefer the linker running faster instead :)
But still, there is more in it than the eye meets.

I have noticed that in code containing a lot of template instantiations (in my case, anything non-trivial using Boost.Spirit) using optimizations can really reduce linker times. My guess is that inlining removes a lot of symbols that the linker would otherwise have to check for duplicates etc. It may be helped by the fact that the linker uses less memory and doesn't need to swap as much.

Offline Ceniza

  • Developer
  • Lives here!
  • *****
  • Posts: 1442
    • CenizaSOFT
Re: Quick&dirty C++ language conformance tests
« Reply #8 on: December 17, 2005, 07:15:24 pm »
Quote from: thomas
And since building wx takes forever...

real    6m26.734s
user    2m59.290s
sys     1m25.821s

I find it quite acceptable, even more when real becomes closer to user :)

Offline me22

  • Official tester
  • Multiple posting newcomer
  • ***
  • Posts: 53
    • TA Universe
Re: Quick&dirty C++ language conformance tests
« Reply #9 on: December 17, 2005, 08:59:04 pm »
I have found this website which provides quick&dirty C++ language conformance tests for the Win32 platform. As C::B supports them, this website could be useful for deciding which compiler is best suitable for a specific project.

Gotta love that of the C++ tests, MinGW ( not even native g++ ) only fails 2, one of which is "throwing destructor" which you shouldn't ever be using anyways.

And what a surprise, the old mingw fails C++0x extension tests *rolls eyes*

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Quick&dirty C++ language conformance tests
« Reply #10 on: December 17, 2005, 09:49:37 pm »
I think this comparison (although still in favour to gcc) is quite unfair. They compare a 3 year old version of gcc with the current release of DM and the current release of Microsoft Visual C++. However, it makes me smile that gcc still beats them.

Yes, I have remarked too that the GCC version was relatively old, but the C++ conformance is a surprise :). I have though that Visual Studio 2005 had a better conformance, especially after all the bla bla of M$. But in the real not :?. I just wonder how could be the conformance of a latest GCC version (and also of the unix GCC original version).

Gotta love that of the C++ tests, MinGW ( not even native g++ ) only fails 2, one of which is "throwing destructor" which you shouldn't ever be using anyways.

And what a surprise, the old mingw fails C++0x extension tests *rolls eyes*

I also like such tests :D. When I have some free time, I search in Internet some comparison between C++ and other languages (recently I have found an interesting comparision/fight between C++ and Fortran. Fortran is much better than C++, C++ is not so OOP, C++ is quite a dead language,... :)).

Anyway, I suppose that C++0x are too new stuff for the old GCC used in the tests. Visual Studio 2005 also do not that better. It shoulds, at least theoretically...Na ja, M$ is full of surprises :).

Michael

Offline Urxae

  • Regular
  • ***
  • Posts: 376
Re: Quick&dirty C++ language conformance tests
« Reply #11 on: December 17, 2005, 10:23:50 pm »
I have found this website which provides quick&dirty C++ language conformance tests for the Win32 platform. As C::B supports them, this website could be useful for deciding which compiler is best suitable for a specific project.

Gotta love that of the C++ tests, MinGW ( not even native g++ ) only fails 2, one of which is "throwing destructor" which you shouldn't ever be using anyways.

Note that at the top he mentions he's evaluating compilers for Win32. Native g++ doesn't qualify, mingw32-g++ does.
That said:
Code: [Select]
D:\Temp>(g++ throwing_destructor.cpp -o throwing_destructor.exe && throwing_destructor.exe && echo Success ) || echo Failure
Success

D:\Temp>(g++ friend_name_injection.cpp -o friend_name_injection.exe && friend_name_injection.exe && echo Success ) || echo Failure
Failure

D:\Temp>g++ --version
g++ (GCC) 3.4.2 (mingw-special)
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(Note: this isn't the latest version, but it's all I have at the moment)
So even though you normally shouldn't be throwing from destructors, by now it's handled correctly if it happens anyway.
Indeed weird that they use MSVC 2005, but such an old MinGW when 2004+ versions are available.

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Quick&dirty C++ language conformance tests
« Reply #12 on: December 18, 2005, 01:36:48 am »
Just for curiosity, I have tried to build the tests on C++ Language Features that failed with MinGW and I could build:

  • throwing destructor
  • friend name injection
  • right angle brackets

The other tests still fail :(. Anyway, there are improvments :).

I used mingw32-g++.exe that come with C::B RC2 (gcc version 3.4.4 (mingw special)).

Michael

Offline Urxae

  • Regular
  • ***
  • Posts: 376
Re: Quick&dirty C++ language conformance tests
« Reply #13 on: December 18, 2005, 02:05:21 am »
Just for curiosity, I have tried to build the tests on C++ Language Features that failed with MinGW and I could build:

  • throwing destructor
  • friend name injection
  • right angle brackets

The other tests still fail :(. Anyway, there are improvments :).

You realize they shouldn't just build but also return a non-zero exit code when you run them, right?

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Quick&dirty C++ language conformance tests
« Reply #14 on: December 18, 2005, 02:52:04 am »
You realize they shouldn't just build but also return a non-zero exit code when you run them, right?

Yes, at least two of them do it. Let me check the missing one.

Michael

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Quick&dirty C++ language conformance tests
« Reply #15 on: December 18, 2005, 03:20:30 am »
Hmmm, the last two of the list return 1, but the first (throwing destructor) returns 0. Visual Studio C++ .NET 2003 and gcc version 3.4.4 (mingw special) gives the same results and behaves (for what I could see) in the same way.

Michael

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: Quick&dirty C++ language conformance tests
« Reply #16 on: December 18, 2005, 04:32:42 am »
-O2 to the linker? 18%? I just tried that and the filesize is still the same.

-Wl,-O2

Am I missing something?

-Os optimizes size
and it doesn't need to be passed directly to the linker.
Still optimizing for speed is more important (-O2, -O3)

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Quick&dirty C++ language conformance tests
« Reply #17 on: December 18, 2005, 03:27:39 pm »
-Os optimizes size
and it doesn't need to be passed directly to the linker.
I can only repeat: read the manual.

Quote
-O level
If  level  is a numeric values greater than zero ld optimizes the output.  This might take significantly longer and therefore
probably should only be enabled for the final binary.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline grv575

  • Official tester
  • Regular
  • ***
  • Posts: 381
Re: Quick&dirty C++ language conformance tests
« Reply #18 on: December 18, 2005, 11:42:25 pm »
Code: [Select]
man gcc says:

-O1 Optimize.
-O2 Optimize even more...performs nearly all supported optimizations that do not involve a space-speed tradeoff.
-O3 Optimize yet more.
-O0 Do not optimize.  This is the default.
-Os Optimize for size.  -Os enables all -O2 optimizations that do not typically increase code size.   It also performs further optimizations designed to reduce code size.

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Quick&dirty C++ language conformance tests
« Reply #19 on: December 19, 2005, 12:01:05 am »
Try man ld  :)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline 280Z28

  • Regular
  • ***
  • Posts: 397
  • *insert unicode here*
Re: Quick&dirty C++ language conformance tests
« Reply #20 on: January 22, 2006, 04:28:07 pm »
If someone tried any of the ones MSVC 2005 fails, I would want to slap them for making me go  :? as I tried to read their code.
78 280Z, "a few bolt-ons" - 12.71@109.04
99 Trans Am, "Daily Driver" - 525rwhp/475rwtq
 Check out The Sam Zone :cool:

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Quick&dirty C++ language conformance tests
« Reply #21 on: January 22, 2006, 05:33:30 pm »
If someone tried any of the ones MSVC 2005 fails, I would want to slap them for making me go  :? as I tried to read their code.

Which one for example?

Michael

Offline 280Z28

  • Regular
  • ***
  • Posts: 397
  • *insert unicode here*
Re: Quick&dirty C++ language conformance tests
« Reply #22 on: January 22, 2006, 05:35:27 pm »
If someone tried any of the ones MSVC 2005 fails, I would want to slap them for making me go  :? as I tried to read their code.

Which one for example?

Michael


Code: [Select]
int main(int argc, char *argv[])
{
  int i = compl((2 bitor 4) xor 4) bitand 3;
  bool b = not ((true or false) and true);

  b and_eq true;
  b or_eq false;
  b xor_eq true;
  b not_eq not b;

  return 0;
}
78 280Z, "a few bolt-ons" - 12.71@109.04
99 Trans Am, "Daily Driver" - 525rwhp/475rwtq
 Check out The Sam Zone :cool:

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Quick&dirty C++ language conformance tests
« Reply #23 on: January 22, 2006, 05:49:33 pm »
Code: [Select]
int main(int argc, char *argv[])
{
  int i = compl((2 bitor 4) xor 4) bitand 3;
  bool b = not ((true or false) and true);

  b and_eq true;
  b or_eq false;
  b xor_eq true;
  b not_eq not b;

  return 0;
}

Yes, I have to admit it has made me confusing too :D. Anyway, I like C++, because it can always surprise me with something I did not even know possible and therefore stimulate my desire to continue studying and using it.

Anyway, e.g., compl is an alternative to the ~ operator. The others are explained here.

Michael

Offline 280Z28

  • Regular
  • ***
  • Posts: 397
  • *insert unicode here*
Re: Quick&dirty C++ language conformance tests
« Reply #24 on: January 22, 2006, 06:16:36 pm »
Yes, I have to admit it has made me confusing too :D. Anyway, I like C++, because it can always surprise me with something I did not even know possible and therefore stimulate my desire to continue studying and using it.

Anyway, e.g., compl is an alternative to the ~ operator. The others are explained here.

Michael


No, no. compl is a way to get coworkers to  :x and then kill you. :lol:
78 280Z, "a few bolt-ons" - 12.71@109.04
99 Trans Am, "Daily Driver" - 525rwhp/475rwtq
 Check out The Sam Zone :cool:

Offline Michael

  • Lives here!
  • ****
  • Posts: 1608
Re: Quick&dirty C++ language conformance tests
« Reply #25 on: January 22, 2006, 06:22:05 pm »
No, no. compl is a way to get coworkers to  :x and then kill you. :lol:

Yes, it is a possible consequence :D.

Michael