Although I am not a big fan of TDD, I am a strong proponent of automated testing. The majority of my experience has been using NUnit, although I have dabbled in Microsoft’s testing framework, MSTest from time to time. I have not used MSTest features that NUnit didn’t already have. Lately I have been hearing more about other unit testing frameworks like MbUnit and xUnit.Net and wanted to put together a high level comparison of the four frameworks to make it easier to decide which to use.
If you are familiar with any of these frameworks and need a quick reference to translate concepts from one framework to another, there is a nice comparison matrix on xUnit’s codeplex site.
The basic premise that most of these unit testing frameworks adhere to is you write a test that asserts some truth (or falsehood) about your code. Related tests are grouped into a test fixture, which allows you to add shared code that runs at the start (setup) and at the end (teardown) of your test execution. In most of the frameworks, test fixtures are classes and tests are methods.
Best practice is to create separate assemblies for your tests. So if you have a class library named “BusinessLogic,” you would create another assembly named “BusinessLogic.Tests” to house all the tests you will write. You can put tests in the same assembly as the classes themselves, but that would bug the heck out of me (it’s only a mild display of OCD).
In the beginning there was NUnit
NUnit was the first mainstream unit testing framework for .Net. It was originally ported from jUnit, a java unit testing framework which is a member of the xUnit (to be confused with xUnit.Net) family of frameworks that originated from SUnit, a Smalltalk unit testing framework.
So after you install the latest version of NUnit, you get a dll that you reference to access the NUnit framework and a graphical test tool which can be pointed to any assembly that has NUnit tests. Besides the core NUnit framework, you may want to install an additional tool like TestDriven.Net for tighter Visual Studio integration. Tools like this make it running and debugging tests trivial.
With NUnit, turning a class into a test fixture is as easy as adding a “[TestFixture]” attribute to your class and “[Test]” attribute to the methods that you want to test. A test will fail if an unexpected exception is thrown or an Assertion fails. Here is an example of a test that leverages NUnit from the NUnit Quickstart:
namespace bank
{
using System;
using NUnit.Framework;
[TestFixture]
public class AccountTest
{
Account source;
Account destination;
[SetUp]
public void Init()
{
source = new Account();
source.Deposit(200.00F);
destination = new Account();
destination.Deposit(150.00F);
}
[Test]
public void TransferFunds()
{
source.TransferFunds(destination, 100.00f);
Assert.AreEqual(250.00F, destination.Balance);
Assert.AreEqual(100.00F, source.Balance);
}
}
}
NUnit is probably the most mature unit testing framework of the bunch. It also has the biggest adoption and community support. There are more articles, discussions, tools and add ons for NUnit than any of the other frameworks combined. Just look at this link on google trends that compares searches for each of the frameworks. No other even comes close.
Reinventing the wheel with MSTest
MSTest made its first appearance with Visual Studio 2005, but only if you have the right sku. If you have VS.Net 2005 Professional or below, you were out of luck. You needed at least Enterprise or Team System Testing Edition to benefit from tight integration with VS.Net. Visual Studio 2008 gets it better, supporting MSTest in all flavors of VS.Net, including the Express editions.
The MSTest attributes are very similar to nUnit attributes. Instead of [TestFixture] you have [TestClass]. Instead of [Test] you have [TestMethod]. Instead of [Setup] you have [TestInitialize]. The Assert functions are even different than NUnit. If anything the MSTest versions are less intuitive than the NUnit attributes. Here is a sample test from a walkthrough I found (after a lot of searching) on msdn:
using Microsoft.VisualStudio.QualityTools.UnitTesting.Framework;
using System;
namespace VSTSDemo.Test
{
[TestClass()]
public class LogonInfoTest
{
[TestInitialize()]
public void Initialize()
{
// TODO: Add test initialization code
}
[TestCleanup()]
public void Cleanup()
{
// TODO: Add test cleanup code
}
[TestMethod]
public void ChangePasswordTest()
{
}
}
}
The main strength of MSTest is that it is bundled with Visual Studio. There will always be some shops out there that will only use stuff that comes from Microsoft, so it will always have a place. Besides that advantage, it also has the ability to generate test stubs for all the methods in a class or set of classes that you specify. This is a pretty neat trick, although its usefulness is somewhat limited in that it only creates one test per method, which is counter to what you typically need to do.
Microsoft’s own Patterns and Practices even acknowledge that there is no real compelling reason to move from NUnit to MSTest. For their flagship product, Enterprise Library, they provide unit tests that use both NUnit and MSTest framework by using compiler directives that allow you to build the tests for either framework.
Getting High with MbUnit
MbUnit, the only testing framework likened to a drug habit, builds on top of the syntax of NUnit, supporting all its behaviors and syntax as well as adding a bunch of useful additional capabilities that differentiate it. It also was built with extensibility in mind and exposes hooks that allow you to control every part of the process.
One of the cooler features of mbunit, is the concept of row tests. Row tests allow you to call a test multiple times with different parameters. Here is an example taken from mbUnit’s own tutorial:
[Row(3)]
[Row(6)]
[RowTest]
public void ToFizzBuzz_SendNumberDivisibleBy3ButNot5_ReturnsFizz(int NumberToTest)
{
Assert.AreEqual("fizz", FizzBuzz.FizzBuzz.ToFizzBuzz(NumberToTest));
}
MStest has a similar capability with the DataSource property, but it is nowhere as clean and requires you to put your data into a database, csv or xml file.
Migrating from NUnit to MbUnit is pretty straightforward, since you will only have to change assembly references, using statements and build scripts.
The only glaring weakness I see in MbUnit are in the documentation and tutorials. I know it has improved recently, but it is still weak in comparison to NUnit. There is also the likely possibility that some of the cooler features will be sucked back into NUnit. Someone already created a Rowtest extension for NUnit.
xUnit.Net - Just Got to be different
The main motivation behind xUnit.Net was to rethink unit testing based on past experiences. In the end, they came up with a new framework that has less than subtle differences with any of the other frameworks. Besides having a completely different syntax, xUnit.Net offers no setup or teardown methods (because they are evil) or expected exception attributes. They also aimed to take advantage of .Net 2.0 features like Generics which make the assertion syntax that much cleaner.
Comparing xUnit.Net to NUnit’s syntax, xUnit.Net has no [TestFixture] attribute, as test methods can be in any class. Instead of [Test] they have [Fact]. Instead of [Setup] you can use a parameterless constructor. Instead of Teardown you can implement IDisposable. They offer similar functionality to mbUnit’s [RowTest], but it is called [Theory]. Here is an example of a xUnit.Net Test from the codeplex tutorial.
public class MyTests
{
[Fact]
public void MyTest()
{
Assert.Equal(4, 2 + 2);
}
}
Besides the fact that xUnit is probably the least mature (current build is a release candidate),the biggest weakness for xUnit.Net is that it is so different from nUnit. Moving from nUnit to xUnit.Net does not look like a straight forward task. I still haven’t used xUnit.Net yet, but I do think some of the concepts, particularly around generics would be useful to have in a unit testing framework.
Wrapping Up
So after my shallow dive into these four unit testing frameworks, I think I need to look a little deeper into MbUnit. If you have any deeper experience with any of these frameworks and can enlighten me into any other advantages or disadvantages, please leave a comment.
