On writing useful (unit) tests

Tests are a means of communication

Although reliably verifying that code performs and continues to perform the intended function is probably the primary reason for writing automated tests, well-structured tests can serve more purposes for developers, most of them boiling down to communication. The disputable Uncle Bob Martin has coined a famous quote in this regard:

(Unit) tests play an important role in guiding a reading developer through a code base. If written with care, they tell stories about the requirements of the system under test, its architecture, and noteworthy corner cases to consider. The test code gives a second perspective on the system and frees readers from analyzing the actual implementation to understand its behavior and requirements. This is only possible if the author of the test cases intended the tests to convey this meaning and to fulfill the goal of guiding others through the code. To my understanding, communicating with test cases is one of the most important aspects about creating useful test code. Similar to code comments, there’s always more knowledge about a system than is told by its direct implementation and tests play an important role in communicating the additional information. Of course, as already noted by Eric Evans, any code should always do its best to tell the correct story:

Efficiently communicating requirements and telling the story of the system that is not or cannot be conveyed by the production code is one of the most important aspects of tests apart from its function as an automated verification.

Tests as a tool for debugging

Another useful benefit of well-written tests is that they can help finding the source of bugs, especially the ones introduced during refactorings or changes to existing code. A badly written test case that fails cannot provide more information than "this huge mess of code doesn’t work anymore after your changes". However, with nicely arranged single-purpose test cases the message of a failing test case is much more granular and helps the developer by saying "most things work, only this special case isn’t handled properly anymore after your refactoring." Such a message makes it much easier to correct the introduced bug and usually avoids a lot of additional debugging work.

Abstraction

A key principle of software engineering is being able to reason about a specific problem by avoiding distractions from lower-level programming details. If I implement a complex algorithm for allocating parcels to carriers, I don’t want to think about network transport errors while finding out the capacity of each carrier in the course of this algorithm. Therefore, a common technique is to abstract from the ugly and unimportant details through a class or interface. Introducing an appropriate abstraction for "find out the carrier capacity" lifts reasoning in the capacity planing algorithm to a single level of …​ abstraction and removes the lower-level networking details from the reasoning process. By following this route, I am – in turn – able to write tests that focus on a single level of abstraction without mixing business problems with infrastructure concerns. Moreover, I can test business (capacity planning) and infrastructure (network communication) concerns in distinct tests. This makes the implementations and the tests easier to formulate and to understand.

Apart from simplifying reasoning, abstraction brings most value if technically realized using interface thinking as already described in 1995 in the classic gang of four book with their "principle of reusable object-oriented design":

That means our abstraction for "find out the carrier capacity" is an interface resembling the level of reasoning required for the capacity planning algorithm and the actual realization with the ugly details of network programming will then be an implementation of this interface:

// The abstraction at the level of the algorithm.
// We are only concerned about requesting capacities for carriers.
public interface CapacityRequestStrategy {
    public int getCapacity(String carrierId);
}

// One implementation of the abstraction
public class NetworkedCapacityRequestStrategy implements CapacityRequestStrategy {
    public int getCapacity(String carrierId) {
        // all the ugly HTTP-details here
    }
}

// The algorithm implementations is free from HTTP details
public class CapacityPlanner {
    private CapacityRequestStrategy requestStrategy = NetworkedCapacityRequestStrategy()

    public void planParcels() {
        while (parcelsRemain) {
            for (String carrier : availableCarriers) {
                // No networking details here!
                final int capacity = requestStrategy.getCapacity(carrier);
                // do some magic to distribute parcels
                // ...
            }
        }
    }
}

Introducing such abstractions avoids having to deal with multiple concerns in the same unit to test and therefore also enables to formulate test cases reflecting the different concerns. Consequently, test become easier, because concerns are separated and each test only deals with a single concern and less cases to look at.

Dependency injection

Given the code shown above, one might ask how to actually write a test for the capacity planning algorithm that doesn’t need mixing abstraction levels. CapacityPlanner still depends on the concrete network-based CapacityRequestStrategy implementation. We would have to employ fancy things like stubbing the HTTP API used to determine capacities to actually test this class, thereby again resorting to mixing abstraction levels. Yuck…​ Fortunately, a cure for this issue is pretty easy: dependency injection. Instead of directly instantiating the NetworkedCapacityRequestStrategy inside the CapacityPlanner, let someone else provide an appropriate instance to the planner by passing it to the planner’s constructor:

public class CapacityPlanner {
    private CapacityRequestStrategy requestStrategy;

    public CapacityPlanner(requestStrategy CapacityRequestStrategy) {
      this.requestStrategy = requestStrategy;
    }


    public void planParcels() {
        // ...
    }
}

Enabling dependency injection on a tested unit opens up the opportunity to install a test double [Fowler2006] inside the automated tests that never has to deal with the complexity of networking:

// This is a test double for the production CapacityRequestStrategy
class ConstantCapacityStrategy implements CapacityRequestStrategy {
    private int capacity;

    public ConstantCapacityStrategy(int capacity) {
        this.capacity = capacity;
    }

    public int getCapacity(String carrierId) {
        return this.capacity;
    }
}

// This test can now be written without having to think about HTTP
class CapacityPlannerTest {
    @Test
    public rejectsParcelsIfNoCapacityRemains() {
        planner = CapacityPlanner(ConstantCapacityStrategy(0));

        assertThrows(
            NoMoreCapacityException.class,
            () -> { planner.planParcels() }
        )
    }
}

Installing test doubles is a real pain without dependency injection, because mocking would be necessary, which is pretty error-prone and usually depends on low-level programming language constructs, thereby bloating test cases with technical details. Moreover, without an appropriate abstraction, the installed test double would again leak details into the algorithm discussions. If an HttpClient were injected instead of an abstraction, I could still install a double, but providing the appropriate behavior in the double would violate the abstraction level suitable for the test and the implementation by resorting to networking again.

Now that the preconditions for being able to write useful tests are met, I can outline my recommendations on how to write the tests themselves.

Verify one aspect per test case

Often, one can find test cases that look like this:

# floating point equality should be handled properly in real code
def test_it_works():
    assert divide(1.0, 1.0) == 1.0
    assert divide(2.0, 1.0) == 2.0
    assert divide(1.0, 2.0) == 0.5
    pytest.raises(ValueError):
        divide(1.0, 0.0)

This test fails as a debugging aid, because the only feedback I get when something is broken is that…​ something is broken. The feedback on the level of failing test cases isn’t more specific, because there’s only on test case that either fails or succeeds as a whole. Wouldn’t it be much better if there was direct feedback from test execution that everything works apart from handling division by zero?

To avoid this trap it is much better to write one test case (function) per tested condition. A much better version with the same assertions could look like:

def test_same_numerator_and_denominator_is_one():
    assert divide(1.0, 1.0) == 1.0

def test_numerator_higher_than_denominator_is_above_one():
    assert divide(2.0, 1.0) == 2.0

def test_numerator_lower_than_denominator_is_below_one():
    assert divide(1.0, 2.0) == 0.5

def test_division_by_zero_is_rejected():
    with pytest.raises(ValueError):
        assert divide(1.0, 0.0)

Now, in case my implementation of divide does the math correctly but I just messed up with the exception handling, test results will immediately tell this picture and I know where to start debugging:

test.py::test_same_numerator_and_denominator_is_one PASSED             [ 25%]
test.py::test_numerator_higher_than_denominator_is_above_one PASSED    [ 50%]
test.py::test_numerator_lower_than_denominator_is_below_one PASSED     [ 75%]
test.py::test_division_by_zero_is_rejected FAILED                      [100%]

Besides providing valuable debugging aids, building test cases per tested aspect also helps communicating requirements on the tested code effectively. I can now use the test case names to communicate what I require from my code to fulfill its technical, and more importantly, business value. Without such test cases, these requirements are often only implicitly represented in the code base. The test code therefore provides additional documentation and explanations that would otherwise be missing, but it cannot become outdated such as comments or external documentation could.

So, in case some of the following conditions match a test function or method, the test case should probably be split:

Use test case names to express readable requirements

As a follow up on the previous rule of using test cases to verify individual requirements, another important aspect for the effective communication of requirements is that they are actually readable as natural language from the code. Humans are much better at understanding natural language than they are at reading highly abbreviated code.

Imagine the division by zero example from above were tested like this:

def test_exception():
    with pytest.raises(ValueError):
        assert divide(1.0, 0.0)

If this test fails, would you know which requirement is currently unmet in case test reports show the following?

test.py::test_exception FAILED                      [100%]

Probably not.

Therefore, use test case names to effectively communicate the imposed requirement on the system under test with natural language. A good test case name includes a verb and can be read as a sentence clearly expressing the verified requirement such as in:

def test_division_by_zero_is_rejected():
    # ...

Some languages such as Kotlin allow making this even more readable by supporting (close to) arbitrary characters in method names:

@Test
fun `division by zero is rejected`() {
    // ...
}

Especially when testing the actual business logic of a system, such a way of naming is of real value, because then business experts or the product owner can understand whether the developers have realized the correct requirements by browsing through the test case names (guided by the developers). In case a requirement was forgotten during implementation, this should become clear to business experts, because they can spot the gap in the natural language specification of what is tested. Therefore, by merely naming test cases correctly, a larger steps towards acceptance testing with value for non-technical stakeholders can be taken. Of course, also the next developer of a system or the future self will value expressive test cases that explain the system and its requirements clearly.

Test your business, not someone else’s

From time to time I’ve stumbled across a pretty interesting testing pattern. Instead of verifying the own requirements, tests were written against framework and language features. For instance, one case in Python looked close to this one:

def my_function(a_param: str) -> int:
    if not isinstance(a_param, str):
        raise ValueError("unsupported type")
    return 42

@pytest.mark.parametrize(
    "param_value",
    [None, 42, 0.0, re.compile(r'.*'), ...]
)
def test_my_function_rejects_other_types(param_value) -> None:
    with pytest.raises(ValueError):
        my_function(param_value)

The affected project had decided to use Python type hints and mypy was used strictly. Therefore, all tooling was set up to write Python close to what a statically typed language would look and feel like. Yet, the test authors somehow repeated a lot of what the tooling was already enforcing using unit tests. Apart from the question where to stop in this specific case (there are indefinitely more types to test here), this approach largely increases the amount of test code to maintain and test runtime increases while creating close to no benefits. You should generally trust the tools you select to an appropriate level or you probably shouldn’t use them. Doing someone else’s work by verifying their implementation shifts a large burden on your own code base that will eventually result in more maintenance work without ever ensuring that your code actually continues to meet its own requirements.

One can say that the previous example at least exercises your own code and verifies that a single conditional works as expected. Even worse are situations like the following one.

class TestMyNetworkedServiceAdapter:
    def test_requests_works(self):
        with pytest.raises(ConnectionError):
            requests.get("http://unknown-host")

Such experiments – most likely used to understand the functioning of an upstream library – remain in the test code surprisingly often. This is really nothing more than code bloat and doesn’t help at all for your own project. So just avoid this.

Of course, there are times when some bug or peculiarity of a used library is causing trouble for your code. But you have probably noticed this because of some missed requirement for your own code. Therefore, whenever possible, try to find a test case that reproduces issues with used tools through special cases of calling your own code. That way these special case tests contribute to the set of requirements imposed by your tests on the system under test and they remain valid even if you later decide to completely drop the buggy dependency. Not having to change test code alongside production code changes is always a good thing and reduces necessary work during refactorings.

Clarify requirements with syntax and features, don’t dilute them

Some test framework provide versatile features for writing tests in concise ways. Especially pytest has accumulated an enormous ecosystem of plugins for solving various (repetitive) tasks through syntactic sugar. The general recommendation with fancy tooling is to use it as a means of improving the documentation quality of test code and not for the sake of applying all plugins that are available. For example, despite using the common feature of parametrizing test cases for reducing the amount of test code, the following test function still weakens the documentation capabilities of the test:

@pytest.mark.parametrize(
    ["numerator", "denominator", "expected_value", "expect_exception"],
    [
        (1.0, 1.0, 1.0, False),
        (2.0, 1.0, 2.0, False),
        (1.0, 2.0, 0.5, False),
        (1.0, 0.0, 0.0, True),
    ]
)
def test_divide(numerator, denominator, expected_value, expect_exception):
    if expect_exception:
        pytest.raises(ValueError):
            divide(numerator, denominator)
    else:
        assert divide(numerator, denominator) == expected_value

While maintaining less code is always something valuable to consider, less code but with higher complexity and lower self-documentation abilities is probably not worth achieving. This version of the tests is still better than the initial test_it_works version, because the test runner reports success and failure for parameter combinations individually and debugging is easier, but the requirements are diluted. Therefore, use such fancy feature for the purpose if making the requirement descriptions stronger, not weaker.

A good example where parametrization is beneficial is to increase coverage within a single requirement:

@pytest.mark.parametrize(
    ["numerator", "denominator"],
    [
        pytest.param(1.0, 1.0, id="basic case"),
        pytest.param(2.0, 2.0, id="ne 1.0 works"),
        pytest.param(-1.0, -1.0, id="both values negative"),
        pytest.param(1.5, 1.5, id="fractional"),
    ]
)
def test_same_numerator_and_denominator_is_one(numerator, denominator):
    assert divide(numerator, denominator) == 1.0

By using named parameters we can even increase the ability of the test cases to explain their exact requirements.