Unit testing tutorial

Assertions

In Google Test, the statements that check whether a condition is true are referred to as assertions. Non-fatal assertions have the EXPECT_ prefix in their names, and assertions that cause fatal failure and abort the execution are named starting with ASSERT_. For example:

TEST (SquareTest /*test suite name*/, PosZeroNeg /*test name*/) {
    EXPECT_EQ (9.0, (3.0*2.0)); // fail, test continues
    ASSERT_EQ (0.0, (0.0));     // success
    ASSERT_EQ (9, (3)*(-3.0));  // fail, test interrupts
    ASSERT_EQ (-9, (-3)*(-3.0));// not executed due to the previous assert
}

Some of the asserts available in Google Test are listed below (in this table, ASSERT_ is given as an example and can be switched with EXPECT_):

Also, Google Test supports predicate assertions which help make output messages more informative. For example, instead of EXPECT_EQ(a, b) you can use a predicate function that checks a and b for equivalency and returns a boolean result. In case of failure, the assertion will print values of the function arguments:

bool IsEq(int a, int b){
    if (a==b) return true;
    else return false;
}

TEST(BasicChecks, TestEq) {
    int a = 0;
    int b = 1;
    EXPECT_EQ(a, b);
    EXPECT_PRED2(IsEq, a, b);
}

Failure
    Value of: b
    Actual: 1
    Expected: a
    Which is: 0

Failure
    IsEq(a, b) evaluates to false, where
    a evaluates to 0
    b evaluates to 1

In EXPECT_PRED2 above, predN is a predicate function with N arguments. Google Test currently supports predicate assertions of arity up to 5.

Fixtures

Google tests that share common objects or subroutines can be grouped into fixtures. Here is how a generalized fixture looks like:

class myTestFixture: public ::testing::test {
    public:
         myTestFixture( ) {
             // initialization;
             // can also be done in SetUp()
         }

    void SetUp( ) {
         // initialization or some code to run before each test
    }

    void TearDown( ) {
         // code to run after each test;
         // can be used instead of a destructor,
         // but exceptions can be handled in this function only
     }

    ~myTestFixture( )  {
         //resources cleanup, no exceptions allowed
    }

     // shared user data
};

When used for a fixture, a TEST() macro should be replaced with TEST_F() to allow the test to access the fixture's members and functions:

TEST_F( myTestFixture, TestName) {/*...*/}

Setting up

There are several options you can choose from when adding Google Test into a CMake project:

• Download the sources and copy them into the project structure.

  This is probably the quickest way to start working with the framework, but there will be no automatic synchronization with the Google Test repository, so you will have to take care of keeping the sources up to date.

  You can find an example of using this approach in the sample project below.

• If you are working with git, you can add gtest as a git submodule for your project.

• Another option is to use CMake to download gtest as part of the build's configuration step.

Checkers

For most of the Boost.Test checkers, you can set a severity level:

• WARN produces a warning message if the check failed, but the error counter isn't increased and the test case continues.

• CHECK reports an error and increases the error counter when the check is failed, but the test case continues.

• REQUIRE is used for reporting fatal errors, when the execution of the test case should be aborted (for example, to check whether an object that will be used later was created successfully).

This way, a Boost checker is usually a macro of the BOOST_[level]_[checkname] format that takes one or several arguments. Basic macros are BOOST_WARN, BOOST_CHECK, and BOOST_REQUIRE. They take one argument of an expression to check, for example:

BOOST_WARN(sizeof(int) == sizeof(long));
BOOST_CHECK( i == 1 );
BOOST_REQUIRE( j > 5 );

A few examples of other checkers are given below:

int i = 2;
int j = 1;
BOOST_CHECK( i == j );     // reports the fact of failure only: "check i == j failed"
BOOST_CHECK_EQUAL( i, j ); // reports "check i == j failed [2 != 1]"

Suites

You can organize Boost tests into suites using the pair of BOOST_AUTO_TEST_SUITE(suite_name) and BOOST_AUTO_TEST_SUITE_END() macros. A simple test suite looks like this:

#define BOOST_TEST_MODULE Suite_example
#include <boost/test/unit_test.hpp>

BOOST_AUTO_TEST_SUITE(TwoTwoFour_suite)
    BOOST_AUTO_TEST_CASE(testPlus) {
         BOOST_CHECK_EQUAL(2+2, 4);
    }
    BOOST_AUTO_TEST_CASE(testMult) {
         BOOST_CHECK_EQUAL(2*2, 4);
    }
BOOST_AUTO_TEST_SUITE_END()

Fixtures

To write a fixture with Boost, you can use either a regular BOOST_AUTO_TEST_CASE macro written after a fixture class declaration or a special BOOST_FIXTURE_TEST_CASE macro:

struct SampleF {
    SampleF() : i(1) { }
    ~SampleF() { }
    int i;
};

BOOST_FIXTURE_TEST_CASE(SampleF_test, SampleF) {
    // accessing i from SampleF directly
    BOOST_CHECK_EQUAL(i, 1);
    BOOST_CHECK_EQUAL(i, 2);
    BOOST_CHECK_EQUAL(i, 3);
}

Setting up

You can choose between three usage variants for the framework: header-only, static library, or shared library. When picking the most suitable option, keep in mind that Boost.Test used as header-only might require significant compilation time.

Find an example of the shared library usage variant in the sample project below (switch to the Boost.Test tab).

Sample test

The following example shows a simple test written with Catch2:

#define CATCH_CONFIG_MAIN // provides main(); this line is required in only one .cpp file
#include "catch_amalgamated.hpp"

int theAnswer() { return 6*9; } // function to be tested

TEST_CASE( "Life, the universe and everything", "[42][theAnswer]" ) {
    REQUIRE(theAnswer() == 42);
}

In the above example, Life, the universe and everything is a free-form test name, which must be unique. The second argument of the TEST_CASE macro is a combination of two tags, [42] and [theAnswer]. Both test name and tags are regular strings that are not limited to be valid C++ identifiers. You can run collections of tests by specifying a wildcarded test name or a tag expression.

Notice the assertion line REQUIRE(theAnswer() == 42). Unlike other frameworks, Catch2 doesn't have a collection of asserts to capture various conditional forms. Instead, it parses the actual C/C++ code of the conditional expression and also uses it to describe the result:

...Failure:
    REQUIRE(theAnswer() == 42)
with expansion:
    54 == 42

The REQUIRE macro aborts a test on failure, while the alternative CHECK macro only reports the failure and lets the test carry on. Within both of these macros, you can use all C++ comparison operators and pass the arguments in any order.

Sections

Another important feature of Catch2 is the way to organize tests in cases and sections (while the class-based fixture mechanism is also supported). Take a look at this example from the documentation:

TEST_CASE( "vectors can be sized and resized", "[vector]" ) {

    // initialization block executed for each section
    std::vector<int> v( 5 );
    REQUIRE( v.size() == 5 );
    REQUIRE( v.capacity() >= 5 );
    // end of initialization block

    SECTION( "resizing bigger changes size and capacity" ) {
         v.resize( 10 );

         REQUIRE( v.size() == 10 );
         REQUIRE( v.capacity() >= 10 );
    }
    SECTION( "resizing smaller changes size but not capacity" ) {
        v.resize( 0 );

        REQUIRE( v.size() == 0 );
        REQUIRE( v.capacity() >= 5 );
    }
}

In the above snippet, TEST_CASE is executed from the start for each SECTION. Two REQUIRE statements at the top of the TEST_CASE enforce that size is 5 and capacity is at least 5 at the entry of each section. This way, shared objects are allocated on stack and there is no need to create a fixture class for them. On each run through a TEST_CASE, Catch2 executes one section and skips the others. Next time, it executes the second section, and so on.

Sections can be nested to create a sequence of checking operations. Each leaf section (a section with no nested sections inside) is executed once. When a parent section fails, it prevents child sections from running. For example:

SECTION( "reserving bigger changes capacity but not size" ) {
    v.reserve( 10 );

    REQUIRE( v.size() == 5 );
    REQUIRE( v.capacity() >= 10 );

    // verify that attempting to reserve a smaller capacity changes nothing
    SECTION( "reserving smaller again does not change capacity" ) {
        v.reserve( 7 );
        REQUIRE( v.capacity() >= 10 );
    }
}

Catch2 also supports the alternative BDD-style syntax for test cases and sections.

Template tests

Catch2 supports type-parametrized test cases in the form of the following macros:

• TEMPLATE_TEST_CASE( test name , tags, type1, type2, ..., typen )

• TEMPLATE_PRODUCT_TEST_CASE( test name , tags, (template-type1, template-type2, ..., template-typen), (template-arg1, template-arg2, ..., template-argm) )

• TEMPLATE_LIST_TEST_CASE( test name, tags, type list )

These macros behave in the same way as regular TEST_CASE, but are run for every type or type combination. See Type-parametrized test cases for details.

In addition to type-parametrized test cases, Catch2 also provides TEMPLATE_TEST_CASE_SIG and TEMPLATE_PRODUCT_TEST_CASE_SIG for creating signature-based parametrized test cases, which have similar syntax with the additional signature argument:

TEMPLATE_TEST_CASE_SIG( test name , tags, signature, type1, type2, ..., typen )

For more information, refer to Signature-based parametrized test cases.

Setting up

Refer to this guide for instructions.

Sample test

A simple test written with Doctest looks like this:

// provides main(); this line is required in only one .cpp file
#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN

#include "doctest.h"

//function to be tested
int fact(int n) {
 return n <= 1 ? n : fact(n - 1) * n;
}

TEST_CASE("testing the factorial function") {
    CHECK(fact(0) == 1); // should fail
    CHECK(fact(1) == 1);
    CHECK(fact(2) == 2);
    CHECK(fact(3) == 6);
    CHECK(fact(10) == 3628800);
}

In the example above, testing the factorial function is a free-form test name, which doesn't have to be unique. CHECK is one of the Doctest's assertion macros.

Test cases, subtests, and suites

Doctest provides the mechanism of subcases for creating nested tests with shared common setup and teardown (however, class-based fixtures are also supported).

TEST_CASE("vectors can be sized and resized") {
    std::vector<int> v(5);

    REQUIRE(v.size() == 5);
    REQUIRE(v.capacity() >= 5);

    SUBCASE("adding to the vector increases it's size") {
        v.push_back(1);

        CHECK(v.size() == 6);
        CHECK(v.capacity() >= 6);
    }
    SUBCASE("reserving increases just the capacity") {
        v.reserve(6);

        CHECK(v.size() == 5);
        CHECK(v.capacity() >= 6);
    }
}

In the code above, TEST_CASE() is executed from the start for each SUBCASE(). Two REQUIRE statements guarantee that size is 5 and capacity is at least 5 at the entry of each subcase. If one of the CHECK()-s fails, this means the test has failed, but the execution continues. On each run of a TEST_CASE(), Doctest executes one subcase and skips the others. Next time, the second one is executed, and so on.

Similarly to Catch's sections, subcases can be nested to create sequences of checking operations. Each leaf subcase (a subcase with no nested subcases inside) is executed once. When a parent subcase fails, it prevents child subcases from running.

You can group test cases into suites using the TEST_SUITE() or TEST_SUITE_BEGIN()/TEST_SUITE_END() macros:

TEST_CASE("") {} // not part of any test suite

TEST_SUITE("math") {
    TEST_CASE("") {} // part of the math test suite
    TEST_CASE("") {} // part of the math test suite
}

TEST_SUITE_BEGIN("utils");

TEST_CASE("") {} // part of the utils test suite

TEST_SUITE_END();

TEST_CASE("") {} // not part of any test suite

Doctest also supports the BDD-style syntax.

Setting up

To start using Doctest, download the latest version of doctest.h and copy it into your project tree. See an example in the sample project below (switch to the Doctest tab).

Quick Documentation for test macros

To help you explore the macros provided by testing frameworks, Quick Documentation pop-up shows the final macro replacement and formats it properly. It also highlights the strings and keywords used in the result substitution:

Show Test List

To reduce the time of initial indexing, CLion uses lazy test detection. It means that tests are excluded from indexing until you open some of the test files or run/debug test configurations. To check which tests are currently detected for your project, call Show Test List from Help | Find Action. Note that calling this action doesn't trigger indexing.