Problem creating same output as VS 2005

[yamokosk]

I am sure you devs already hear this enough but.. wow, CB is amazing. I finally gave up on VS 2005 after being annoyed to no end by countless hang ups and a very, very sluggish intellijoke system. Great work guys! Keep it up.

However, I have run into a little bit of a snag. I have been using VS 2005 to compile Matlab MEX files (really just a DLL with a special exported function - mexFunction). But I can't seem to get CB to properly build these same MEX projects. To help debug this problem, I created a simple "Hello World" MEX project in Visual Studio. When I compile the library and then dumpbin the contents I get:

Microsoft (R) COFF/PE Dumper Version 8.00.50727.762
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file SimpleMexVS.mexw32

File Type: DLL

  Section contains the following exports for SimpleMex.dll

    00000000 characteristics
    467A8F0C time date stamp Thu Jun 21 10:45:32 2007
        0.00 version
           1 ordinal base
           1 number of functions
           1 number of names

    ordinal hint RVA      name

          1    0 0001101E mexFunction = @ILT+25(_mexFunction)

  Summary

        1000 .data
        1000 .idata
        2000 .rdata
        1000 .reloc
        1000 .rsrc
        4000 .text
       10000 .textbss

Then in CB (nightly build, Jun 18, 2007) I have created an identical project (DLL target, using VS 2005 compiler, same source code, and meticulously copied every option from VS to CB). But when I compile in CB then dumpbin the contents of the library I get something different:

Microsoft (R) COFF/PE Dumper Version 8.00.50727.762
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file SimpleMexCB.mexw32

File Type: DLL

  Section contains the following exports for SimpleMex.dll

    00000000 characteristics
    467AA204 time date stamp Thu Jun 21 12:06:28 2007
        0.00 version
           1 ordinal base
           1 number of functions
           1 number of names

    ordinal hint RVA      name

          1    0 00001005 mexFunction = @ILT+0(_mexFunction)

  Summary

        1000 .data
        1000 .idata
        2000 .rdata
        1000 .reloc
        1000 .rsrc
        4000 .text

This difference is important because when I try to load each respective function in MATLAB I get an error from the CB compiled version. Anyone have any ideas where to proceed. As I said, I have double checked that all compile/linker options, preprocessor symbols, etc are the same between the VS and CB projects. Are there any other debugging ideas anyone can think of?

[mandrav]

Morten us building matlab dlls with C::B but he's not using the VC compiler. He's using LCC or GCC.
Maybe he has something to share...

[MortenMacFly]

Maybe he has something to share...

I have. ;-)
Well - I can't really help you with VC2005 as I don't have it. But for this case I think you are still missing some compiler flags. Have a look at the mex batch file that is used (setup) for your compiler... (I assume you know the interna of the mex build process). This was my starting point, too when it came to what compiler/linker flags to use. In addition it could be that you are linking against the wron MSVC lib - remember that Matlab ships with several one, depending on the VC compiler you are using. Hopefully you didn-'t forget to link against the required Matlab libs, too... but this should throw a linker error otherwise.

I am using the Matlab built-in LCC (I have a special version of C::B for that - including support for that compiler) or the MinGW/GCC. The latter is not easy to configure, but it is possible. I can guide you through it if you like. If you need the VC compiler (I can imagine several reason why this could be the case) try to compile with another version in addition - VC6, if this is available to you. Search the C::B WiKi for information how to setup this compiler in C::B. I only have that compiler so I could help you better then... maybe... ;-)

With regards, Morten.

[yamokosk]

Morten-

Good point about checking the Matlab options file for my compiler. There was one or two options listed in it that neither VS nor C::B contained. But adding them to C::B did not remedy the situation.

Here is what I don't understand. C::B is merely an IDE. Under the covers, it is doing everything VS is doing.. calling cl.exe + my options, calling link.exe + my options, etc. And if I can verify that "my options" are the same between both programs (which I am guessing they are not), then they should both produce identical output.

Is there any way to capture the command terminal steps of both programs? So I can see what is actually being passed to cl.exe by VS and cl.exe by C::B?

(btw, if anyone was wondering I did track down one of the differences in the dumpbin's between the two compiled libraries. From MS documentation, "Section .textbss is only required by Microsoft Visual C++ 7.0 and higher if incremental linking is used." I verified that /INCREMENTAL is a linker option in both C::B and VS... But the C::B dumpbin does not contain a .textbss section. Is the /INCREMENTAL option not getting passed to link.exe in C::B?)

[mandrav]

Is there any way to capture the command terminal steps of both programs? So I can see what is actually being passed to cl.exe by VS and cl.exe by C::B?

I can't help you with VS (as I don't have it), but for C::B do this:
Go to "Settings->Compiler and debugger", select your compiler from the drop-down list at the top and then click on the "Other" page. There, select the logging option and set it to "Full logging". This will print the full build commands, as executed.

[TDragon]

Is there any way to capture the command terminal steps of both programs? So I can see what is actually being passed to cl.exe by VS and cl.exe by C::B?

VS2005 generally creates a build log in the intermediate files directory called BuildLog.htm. In Code::Blocks, you can see the full command line in the Build Log window by changing the Compiler logging option in Settings->Compiler and debugger->Global compiler settings->Other settings to "Full command line".

Edit: Beatz0r'ed.

[MortenMacFly]

Is there any way to capture the command terminal steps of both programs? So I can see what is actually being passed to cl.exe by VS and cl.exe by C::B?

In Matlab there is a verbose switch (or similar) for the compiler... I don't recall exactly - but I can have a look when I'm at work if you like. For C::B have a look at my sig. It will tell you what to do... ;-)

And yes: If the command line produced by C::B is the same and the one from Matlab AND the path settings are correct (e.g. consider environment variables, too maybe) then it *will* produce identical output (besides date stamps maybe). For sure.

With regards, Morten.

[MortenMacFly]

VS2005 generally creates a build log in the intermediate files directory called BuildLog.htm.

I believe he isn't compiling in VC2005 but in Matlab - but even there *could* be created a build log... if it is setup (which is the default IMHO).
With regards, Morten.

[yamokosk]

I am moving from VS to C::B.. well to be more specific it was more like Matlab > VS > C::B. Just to review, I can compile and run mex files from VS with no problems. Its only when I try and do the same thing in C::B that I am running into trouble.

At any rate, I found that build log produced by VS (I wish it was more of an integrated option like it is for C::B..). But at any rate, I will pour over the two logs to see whats missing or not getting included.

[MortenMacFly]

Just to review, I can compile and run mex files from VS with no problems.

Oh! Ok... this was my misunderstanding. I though you only use the compiler of VS2005 but compile inside Matlab. (I always did that.) Ok then... probably in this case the batch file from Matlab is incorrect. Make sure you do a "mex -setup" before (on the Matlab prompt) and configure Matlab appropriately for the VC compiler. But you can also parse the log... this should contain the most important information, too.
With regards, Morten.

[yamokosk]

Ok, I did a lot of testing and discovered two things: one possible bug in C::B and the other is a major difference in the tool paths between C::B and VS.

First the possible bug. After thoroughly scouring the build logs of both VS and C::B, I found that the link option "/DEBUG" was present in VS but not in C::B. To try and fix this, I went to C::B->Build Options->Link Settings->Other linker options and added /DEBUG. But whenever I hit "Ok" and recompiled, the "/DEBUG" option did not show up. In fact when I went back to the build options page, what I had entered disappeared?! Maybe this has already been reported? Anways, as a quick fix, I went to Settings->Compiler and debugger...->Other settings->Advanced options... and essentially hardcoded "/DEBUG" into the "Link object files to dynamic library" command. After the fix, what do you know.. The dumpbin for both the C::B compiled library and the C::B compiled mex file were identical!

But that still did not completely fix my problems. I am not a C++/Windows/VS guru but what is a manifest? This seems to be where the C::B tool path differs significantly from VS. More specifically, when I instruct VS to not "Embed the Manifest", I get the same bad mex file produced by C::B (aka Matlab gives the same error message for both mex files). But when I embed the manifest in VS, I get a working mex file.

Also one other observation. VS allows options to be specified for the resource compiler tool. Is this functionality in the works for a future release of C::B? What about this manifest thing?

[Biplab]

The problem is wizard doesn't setup a Post-build Job to embed manifests. I believe this is the root cause.

MSVC 8 (MSVC 2005) requires a special manifest file (An XML file) to load debug CRT. As you're compiling DLL in debug mode, it needs that manifest file either in embedded form or in the same directory where your DLL resides.

I'm updating the wizard so that a post-build job gets executed to embed the manifest file.

First the possible bug. After thoroughly scouring the build logs of both VS and C::B, I found that the link option "/DEBUG" was present in VS but not in C::B. To try and fix this, I went to C::B->Build Options->Link Settings->Other linker options and added /DEBUG. But whenever I hit "Ok" and recompiled, the "/DEBUG" option did not show up. In fact when I went back to the build options page, what I had entered disappeared?! Maybe this has already been reported? Anways, as a quick fix, I went to Settings->Compiler and debugger...->Other settings->Advanced options... and essentially hardcoded "/DEBUG" into the "Link object files to dynamic library" command. After the fix, what do you know.. The dumpbin for both the C::B compiled library and the C::B compiled mex file were identical!

This is the full commandline which C::B generates and it has /DEBUG directive.

Code

link.exe /dll /nologo /LIBPATH:"C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2\lib" /LIBPATH:"C:\Program Files\Microsoft Visual Studio 8\VC\lib"  /out:bin\Debug\TestDLL.dll user32.lib  obj\Debug\main.obj   /DEBUG

[MortenMacFly]

I went to C::B->Build Options->Link Settings->Other linker options and added /DEBUG. But whenever I hit "Ok" and recompiled, the "/DEBUG" option did not show up.

Sure not. If you enter such option by hand that is part of the "official" list of compiler/linker settings it will be "transferred" into the appropriate checkbox being activated. It works like that:
- parse the list of compiler/linker options
- if it's found in the "official" list -> activate the appropriate checkbox
- if it cannot be found add it to the text field "just as it is"
So in your case if you added "/DEBUG" by hand int he text field next time you enter the dialog the appropriate checkbox is activated -> the settings do not get lost.
With regards, Morten.

[Biplab]

The updated wizard is in Rev 4153. So wait for todays (or tonight's ) nightly. It should solve your problem.

Please note that in the new wizard, incremental linking is set to NO. This is necessary to embed the manifest in a simple way.

[yamokosk]

The updated wizard is in Rev 4153.

Great! I will give the new build a test run when it comes out.

So in your case if you added "/DEBUG" by hand int he text field next time you enter the dialog the appropriate checkbox is activated -> the settings do not get lost.

This is the full commandline which C::B generates and it has /DEBUG directive.

Maybe I overlooked something but I am almost positive that the full command line that C::B generated did not have the /DEBUG option in it. That was after adding it to the extra options list about 3 or 4 times... but if you say its there, then maybe it was a user error. Morten - Thanks for the explanation of the disappearing options. I had a feeling something like that was going on behind the scenes because I witnessed similar behavior for other options.

Please note that in the new wizard, incremental linking is set to NO. This is necessary to embed the manifest in a simple way.

This is no problem for my project. I toggled this option in VS and Matlab didn't seem to care either way.. which makes sense I guess.

[yamokosk]

Great! I will give the new build a test run when it comes out.

Woohoo! It works. You guys rock!... I tried out build 4177 and it compiles the Matlab Mex functions (on Windows, using VS2005) just fine now! The manifest thingy fixed it. I just had to change some of the directories the mt.exe program was looking in to find some of my files but otherwise it works great.

[Biplab]

I just had to change some of the directories the mt.exe program was looking in to find some of my files but otherwise it works great.

Is it a bug or a customisation that you needed? Please post if wizard created project doesn't work well.

[yamokosk]

Is it a bug or a customisation that you needed? Please post if wizard created project doesn't work well.

No I don't think its a bug. I just have my directory structure set up a little differently:

/src/*.cpp
/include/*.h
/bin/debug
/bin/release
/obj/debug
/obj/release
/compile/*.cdb

And also, MATLAB likes their compiled mex functions to have a different extension than DLL. Instead they want *.mexw32. I could probably make a wizard just for Matlab mex functions?

About my directory structure, are their any built-in variables in C::B which could be substituted for the hard coded values used in the wizard?

[MortenMacFly]

I could probably make a wizard just for Matlab mex functions?

It's not as easy as you may think - you will have to take a lot into account. There are at least 3 different Matlab versions (in terms of MEX files) that require different settings which itself depend on the compiler. In addition the language matters - you can have Fortran, C and C++ MEX functions. BTW: the mexw32 extension is rather new so this would break any old Matlab version - so be careful with this. BUT: I am working on that, too already. It's not yet perfect but it's btw the most complex wizard I've ever done. ;-) Any yes: It *has* to be a different wizard than the DLL one.
So, feel free to contribute, I would merge our work then before release.
Hence I wouldn't commit such into SVN unless we have added support for LCC which is the default compiler for Matlab. Everything else would be just too confusing for Matlab newbies IMHO... That's at least my experience form work.
With regards, Morten.