Catch

Catch2 is a light-weight testing framework. The name stands for C++ Automated Test Cases in Headers (version two). CLion supports Catch versions 1.7.2 and later.

As well as Boost.Test, Catch2 doesn't provide mocking functionality. However, you can combine it with standalone mocking frameworks such as Hippomocks, FakeIt, or Trompeloeil.

Catch2 basics

If you are not familiar with Catch/Catch2, you can find a description of its main concepts below:

Sample test

The following example shows a simple test written with Catch2:

                    #define CATCH_CONFIG_MAIN // provides main(); this line is required in only one .cpp file
                    #include "catch_amalgamated.hpp"

                    int theAnswer() { return 6*9; } // function to be tested

                    TEST_CASE( "Life, the universe and everything", "[42][theAnswer]" ) {
                    REQUIRE(theAnswer() == 42);
                    }
                

In the above example, Life, the universe and everything is a free-form test name, which must be unique. The second argument of the TEST_CASE macro is a combination of two tags, [42] and [theAnswer]. Both test name and tags are regular strings that are not limited to be valid C++ identifiers. You can run collections of tests by specifying a wildcarded test name or a tag expression.

Notice the assertion line REQUIRE(theAnswer() == 42). Unlike other frameworks, Catch2 doesn't have a collection of asserts to capture various conditional forms. Instead, it parses the actual C/C++ code of the conditional expression and also uses it to describe the result:

                        ...Failure:
                        REQUIRE(theAnswer() == 42)
                        with expansion:
                        54 == 42

The REQUIRE macro aborts a test on failure, while the alternative CHECK macro only reports the failure and lets the test carry on. Within both of these macros, you can use all C++ comparison operators and pass the arguments in any order.

Sections

Another important feature of Catch2 is the way to organize tests in cases and sections (while the class-based fixture mechanism is also supported). Take a look at this example from the documentation:

                        TEST_CASE( "vectors can be sized and resized", "[vector]" ) {

                        // initialization block executed for each section
                        std::vector<int> v( 5 );
                        REQUIRE( v.size() == 5 );
                        REQUIRE( v.capacity() >= 5 );
                        // end of initialization block

                        SECTION( "resizing bigger changes size and capacity" ) {
                        v.resize( 10 );

                        REQUIRE( v.size() == 10 );
                        REQUIRE( v.capacity() >= 10 );
                        }
                        SECTION( "resizing smaller changes size but not capacity" ) {
                        v.resize( 0 );

                        REQUIRE( v.size() == 0 );
                        REQUIRE( v.capacity() >= 5 );
                        }
                        }
                    

In the above snippet, TEST_CASE is executed from the start for each SECTION. Two REQUIRE statements at the top of the TEST_CASE enforce that size is 5 and capacity is at least 5 at the entry of each section. This way, shared objects are allocated on stack and there is no need to create a fixture class for them. On each run through a TEST_CASE, Catch2 executes one section and skips the others. Next time, it executes the second section, and so on.

Sections can be nested to create a sequence of checking operations. Each leaf section (a section with no nested sections inside) is executed once. When a parent section fails, it prevents child sections from running. For example:

                        SECTION( "reserving bigger changes capacity but not size" ) {
                        v.reserve( 10 );

                        REQUIRE( v.size() == 5 );
                        REQUIRE( v.capacity() >= 10 );

                        // verify that attempting to reserve a smaller capacity changes nothing
                        SECTION( "reserving smaller again does not change capacity" ) {
                        v.reserve( 7 );
                        REQUIRE( v.capacity() >= 10 );
                        }
                        }
                    

Catch2 also supports the alternative BDD-style syntax for test cases and sections.

Template tests

Catch2 supports type-parametrized test cases in the form of the following macros:

TEMPLATE_TEST_CASE( test name , tags, type1, type2, ..., typen )
TEMPLATE_PRODUCT_TEST_CASE( test name , tags, (template-type1, template-type2, ..., template-typen), (template-arg1, template-arg2, ..., template-argm) )
TEMPLATE_LIST_TEST_CASE( test name, tags, type list )

These macros behave in the same way as regular TEST_CASE, but are run for every type or type combination. For more information, refer to Type-parametrized test cases.

In addition to type-parametrized test cases, Catch2 also provides TEMPLATE_TEST_CASE_SIG and TEMPLATE_PRODUCT_TEST_CASE_SIG for creating signature-based parametrized test cases, which have similar syntax with the additional signature argument:

TEMPLATE_TEST_CASE_SIG( test name , tags, signature, type1, type2, ..., typen )

For more information, refer to Signature-based parametrized test cases.

Working with Catch2 in CLion

Add Catch2 to your project

Follow the instructions in this guide to start with Catch using its CMake integration.

Run/debug the automatically created Catch configuration

CLion detects the Catch tests in your project and creates a run/debug configuration for them.

You can use this configuration right away to run or debug all the Catch tests in your project:

Add a new Catch run/debug configuration

Go to Run | Edit Configurations.
Click and select Catch from the list of templates.
- Set the configuration name in the Name field. This name will be shown in the list of the available run/debug configurations.
- Select the Tags/Test option to run a test for the particular tags or all the tags. Select the Pattern option to run all the tests for a particular pattern.
- Specify tags in the Tags fields. This field is available only when the Tags/Test option is selected.
- Select the desired test from the Test list. This option is available only when one or more tags have been provided.
- Specify the pattern name in the Pattern field. This field is available only when the Pattern option is selected.
- In the Target field, select the desired target from the list of available targets.
Instead of editing a single configuration, you can modify the Catch template itself. In this case, the settings you specify will apply as defaults to all new configurations of this type.
Click Apply to save the configuration.
Select the newly created configuration in the switcher and run or debug it.

Run tests

The quickest way to run or debug a single test or a whole suite/fixture is by using the gutter icons:
Gutter icons also show test results (when already available): success or failure .
Refer to Run, debug, terminate tests for more information.
When you run a test/suite/fixture using gutter icons, CLion creates a temporary Catch configuration, which is greyed out in the list of configurations. To save a temporary configuration, select it in the Edit Configurations dialog and click :

Explore test results

When you run tests, CLion shows the results and the process in the built-in test runner window. The test tree shows all the tests while they are being executed one by one. The test runner window includes:

progress bar with the percentage of tests executed so far,
tree view of all the running tests with their status and duration,
tests' output stream,
toolbar with the options to rerun failed tests, import/export or open previous results saved automatically , sort the tests alphabetically to easily find a particular test, or sort them by duration to understand which test ran longer than others.

Specify random seed

If you are using randomization in your Catch tests, you can set the initial seed by specifying the --rng-seed='time'|'random-device'|'number' parameter. See this section in the Catch documentation for details.

In the main menu, go to Run | Edit Configurations and select any Catch configuration. The seed parameter will be used for the entire test run.
In the Program arguments field, set the --rng-seed flag:
Run the configuration, and you will see the fixed seed in the test tree:
Use the context menu to rerun the tests with the same fixed seed:
Do not use , , or configurations to rerun the tests if you want to keep the previously generated seed.

Troubleshooting: long test names produce incorrect listings and terminate test run

If there are long test names in your project, this may break the listing and terminate the whole test run (see this Catch issue). Try one of the following solutions:

Open the catch_user_config.hpp file and change the #define CATCH_CONFIG_CONSOLE_WIDTH parameter from 80 to 800.
Open the CatchConfigOptions.cmake file and change the config generator from
set(CATCH_CONFIG_CONSOLE_WIDTH "80" ...)
to
set(CATCH_CONFIG_CONSOLE_WIDTH "800" ...)
(Least recommended) Disable the test list extraction in CLion: clear the cidr.test.prepare.test.list registry key. Use Help | Find Action Ctrl+Shift+A to open the Registy dialog.

Last modified: 08 October 2024