The Three Characteristics of Automated Tests That Actually Matter

Automated Tests are important. People know this. But do they really know what matters when it comes to automated testing?

Well, let’s think about it. What are we trying to accomplish with our automated tests? We’re trying to make sure that the software we just wrote functions and that we didn’t break any existing functionality. But we can do that with manual tests. So what makes automated tests special? The main thing that makes them special is they are automated.

Automated Tests Must Actually Be Automated

I know this seems obvious, but you’d be surprised how easy it is to get this one wrong. For example, one place I worked had a suite of automated tests that had to be manually deployed (by a person), started (by a person), and then monitored (again, by a person). Not very automated, after all.

Automated Tests Must Actually Test the Program

Code coverage and relevance are probably some of the hardest testing concepts to get right. Wanna know why? Because there’s not a definite right answer. The normal target number for code coverage is 80%, but if you’re writing tests just to cover lines and not actually exercise functionality then that number is meaningless. Relevance is more critical than coverage, but there’s no objective way to really measure it. Just making sure you have tests for all the requirements or acceptance criteria is probably a good place to start. After that, just target important sections and give them some extra love.

Automated Tests Must Be Deterministic

If one of your solutions to test failures is, ” run it again,” then you have a problem here. If I give the same test the same inputs, then i should get the same results every time. Automated tests should only fail because the code is bad, not because the tests are bad.

What About Unit vs. Non-unit Tests

Unit tests are good because it’s easy to make them fit all the criteria. But if you can have integrated tests or functional tests that fit the criteria, then use them. In fact, it would probably be a good strategy to mix them all in together. The more tests, the better. Don’t worry about whether or not you’ve truly isolated your test to a unit, worry instead that your code will work, and if it doesn’t then a test will tell you.

So, get out there and write some tests. Hopefully, they’re automated. But if not, write them anyway. Just write some.

Mocking Without a Mocking Framework

In C#, because of delegates and code blocks, it turns out we don’t really need mocking frameworks. We can easily create mock objects that will let you define and monitor behavior on the fly. It’s kind of like a mocking framework, but without 3rd party dependencies and licensing costs.

For example, let’s take a sample repository class. We need a mock of it so that we can unit test a controller. Here’s the interface:

public interface IBookRepository
{
    List<Book> GetBookList();
    Book GetBook(Guid id);
    Book AddBook(Book newBook);
    Book UpdateBook(Guid id, Book updatedBook);
    bool DeleteBook(Guid id);
}

We’re going to create a mock object that implements this interface in a way that my unit tests can define the behavior as needed. In the unit test project, I created a Mocks folder and then added a new class to that folder, MockBookRepository

internal class MockBookRepository : IBookRepository

The next step is to create function delegates for the methods that need to be implemented:

public Func<List<Book>> MockGetBookList { private get; set; }
public Func<Guid, Book> MockGetBook { private get; set; }
public Func<Book, Book> MockAddBook { private get; set; }
public Func<Guid, Book, Book> MockUpdateBook { private get; set; }
public Func<Guid, bool> { private get; set; }

Now we implement the actual methods with the following strategy: if the matching delegate is defined, run it, otherwise do nothing. Here’s an example of one of the methods:

public Book GetBook(Guid id)
{
    return MockGetBook != null ? MockGetBook(id) : null;
}

I thought about throwing an exception here if the delegate wasn’t defined, but it makes more sense to have the unit test determine if that is actually undesired behavior. Note that if the return type is not nullable, then we can’t return null like the example. In those cases, just return the default value of that type. When it’s all written, we have a class that does absolutely nothing until its behavior is defined. The whole class looks like this:

internal class MockBookRepository : IBookRepository
{
    public Func<List<Book>> MockGetBookList { private get; set; }
    public Func<Guid, Book> MockGetBook { private get; set; }
    public Func<Book, Book> MockAddBook { private get; set; }
    public Func<Guid, Book, Book> MockUpdateBook { private get; set; }
    public Func<Guid, bool> { private get; set; }

    public Book GetBookList()
    {
        return MockGetBookList != null ? MockGetBookList() : null;
    }

    public Book GetBook(Guid id)
    {
        return MockGetBook != null ? MockGetBook(id) : null;
    }

    public Book AddBook(Book book)
    {
        return MockAddBook != null ? MockAddBook(book) : null;
    }

    public Book UpdateBook(Guid id, Book book)
    {
        return MockUpdateBook != null ? MockUpdateBook(id, book) : null;
    }

    public bool DeleteBook(Guid Id)
    {
        return MockDeleteBook != null ? MockDeleteBook(id) : false;
    }
}

Now all that’s left is to use it in the controller test. If all I want is an object that can be called without throwing exceptions, all I have to do is define it.

var bookRepository = new MockBookRepository();
var controller = new BookController(bookRepository);

But for this test, I actually want the GetBook(Guid) method to return a book. So I define that on the fly in my unit test.

var bookRepository = new MockBookRepository
{
    MockGetBook = id => new Book {Id = id, Title = "Royal Assassin", Author = "Hobb, Robin"}
};

So far, so good. But now I need to make sure the method on the repository was called. All I need to do is define a boolean in my unit test and set it to true in my delegate definition.

var getBookWasCalled = false;
var bookRepository = new MockBookRepository
{
    MockGetBook = id =>
    {
        getBookWasCalled = true;
        return new Book {Id = id, Title = "Royal Assassin", Author = "Hobb, Robin"};
    }
};

Now if I Assert that getBookWasCalled is true, then I know it was called. Notice that the definition changed to a normal code block. You can do pretty much anything you want inside that code block, which is where this method suddenly becomes a lot more powerful and intuitive than any mocking framework. With this wide open code, you can do a lot of things:

  • Count how many times a method was called
  • Verify the parameters it was called with
  • Do different things with different inputs
  • Throw an exception if it wasn’t supposed to be called
  • Ignore the parameters it was called with
  • Throw an event
  • etc.

Anything that any framework can do can be easily coded in the code block.