Unit Test Plugin

[koonschi]

Hello,

I'm currently writing a UnitTest plugin based on UnitTest++. There are already means to execute unit tests by adding them as a post build command.
While this works fine most of the time, I was still unsatisfied with it.

The motivation was to start unit tests from the IDE, like, for example JUnit or VS are doing it. Basically, right-click into the editor "UnitTest" -> "Run current test" or "Run current test suite".

It's meant to work like this when it's done:
The user creates a Test Project. This test project has a predefined main.cpp, which parses command line options and then runs tests based on the input from the command line.
This main.cpp is not meant to be manipulated (but can be nevertheless ofc).
Then you only have to add new files with tests and test suites a la UnitTest++. You can either run them all at once (simply build & run the project), or you can run them selectively from codeblocks.
I also want some kind of test list, where you can see what tests or suites there are and then run them from there. I'm basically thinking of something similar to the VisualStudio 2010 interface.

I've been using UnitTest++ for quite some time now and I've made a few adjustments to it over time, in order to report better error messages, more means for checking values or for comparing stl containers.

What I got to work so far:

• RMB -> Unit Testing -> Run Test
• RMB -> Unit Testing -> Run Suite
• RMB -> Unit Testing -> Run All Tests
• Running tests compiles the test program first
• Test output gets displayed & parsed by the plugin (similar to compilation output)
• Double click on a test failure jumps to the file + line where it failed

What's still to do:

• Logging of test results
• List containing all tests of one or more projects
• List with failed or successful tests
• Code Coverage (gcov)
• Comfort functions like 'create a new test file'
• Polishing

Screenshots so far:
http://i.imgur.com/zhuYek7.png
http://i.imgur.com/Ecytzof.png
http://i.imgur.com/gNXcGya.png

My biggest concern is, that the plugin requires the adjusted version of the UnitTest++ framework, because a test project of the plugin must be based on the UnitTest++ framework. Otherwise the plugin can't parse the Tests and Suites or the output.
Would it work if the plugin simply creates a test project that already contains, say, 25 files (== the UnitTest++ files)?

I'd be happy to hear what you think.

Edit: Attached source.

[oBFusCATed]

I've not read it all, but I'm sure I don't like it because:
1. the right click menu is already too cluttered and you're adding stuff too it
2. the same for logs&messages
3. unittest++ only dependency (I'm using it, but there are plenty of other testing frameworks that many people use)
4. the project automagic will break too easily

[stahta01]

FYI: If you plan to have it added to the CB Official Contrib, at current time the CB Team is NOT doing C++11 needed for override and for range.

Attached a windows cbp project; has untested Unix part.

Tim S.

[koonschi]

1. the right click menu is already too cluttered and you're adding stuff too it
2. the same for logs&messages

I agree with you, and I'm planning to only let the right click menu appear when you right click into a file that contains tests.
If you have a better idea for displaying test output, I'd be happy to hear it. For now this was the quickest solution.

3. unittest++ only dependency (I'm using it, but there are plenty of other testing frameworks that many people use)

I'm aware of that, but I think the lack of integrated unit test support is quite a big issue.

4. the project automagic will break too easily 

I think I don't understand?

[killerbot]

the log and messages is ok for me. (see remark at the end)

But the right click menu is indeed to cluttered, ideas :

- submenu in Build menu ( or project menu ?)
- a dedicated menu (next to (right hand side) Debug)

- colors : green and red : number of tests, failures / success / ...

Indeed in such a case one wants to have the unit tests "library" present. I think "according to the licensing" we might bundle the sources and have the lib as part of CB.
It is a small a nice framework, good thing to start with and have something in the IDE. Once we have something we could make it more generic, one can think of cppunit, google test etc ...
But step by step ...


If we think we have too much messages, then we should :
- build log / build messages
- valgrind log valgrind messages
- cppcheck log / cppcheck messages
- unit test log / unit tests messages
...
The nthey all should go to one instance of :
- foo log / foo messages
where I have no clue yet on how to call 'foo' ;-)

[koonschi]

Progress so far:

I created a menu bar, a toolbar and an additional option "repeat last run". This comes in handy if you're editing a file and just want to repeat the test to see whether it worked.
Furthermore, the right-click menu is only shown if tests are detected in the current file, otherwise it's left alone. I think this is ok, isn't it?

I have two problems right now:
First: How could I 'mark' a project as a test project? I was thinking of either some kind of setting in the project file (which needs to be saved, too). If you have a better idea, I'd be happy to hear it.
Second: How do I **reliably** enter the add the menu bar entry after Debug? I have a feeling that sometimes my plugin is loaded before the Debug entry exists, thus placing it wrong.
 
Progress Screenshot:
http://i.imgur.com/cPKrgEB.png

[killerbot]

naive approach : unit test project when linked with unittest++ ?

[koonschi]

Could work. I'll give it some more thought.

[koonschi]

I've come up with an even simpler method. I have to detect all tests in all open projects anyway, so I'll just assume a project is a test project if it contains tests .

But there is still the problem with the menu bar. If the Unit Testing menu is supposed to be after the Debug menu, how can I reliably insert it there?
Looking for the Debug menu and then inserting my entry after it doesn't work.

[oBFusCATed]

The debug menu is created after the first debugger plugin is loaded. If there are no debugger plugins there will be no debug menu.

[koonschi]

The error occurs only on startup. When I install the plugin over the menu everything works fine.

Is it possible that my plugin might be loaded before the debugger plugin when c::b starts? If yes, how do I then create the menu entry at the right position?

[oBFusCATed]

Try to not depend on the presens of the Debug menu. If the Debug menu is not there just create it after the Build and if Build is not present create it after Project. Unfortunately there is no more robust way to implement this.

[Pecan]

You also have cbEVT_PLUGIN_LOADING_COMPLETE event available.
You should be able to modify any menu entry at that time.
    
Example:
Manager::Get()->RegisterEventSink(cbEVT_PLUGIN_LOADING_COMPLETE, new Event(this, DebuggerManager::OnPluginLoadingComplete));

[Alpha]

I remember writing some sort of delayed menu building in my (sort of) abandoned plugin here.  (It worked then, but I have not looked at it recently, so no guarantees.)