The new release of cmake contains a feature I have been waiting on for like a decade or more. An easy way to build your unit tests, and nothing more, just before you want to invoke ctest to run them.
Situation/Environment
A codebase typically consists out of many libraries, and many executables. The latter depend on a bunch of libraries, which in turn depend on another set of libraries which in turn …
Several of those libraries and executables have unit tests, others have not.
A good CI, builds the entire codebase (preferably in an incremental way) to ensure that with each commit, everything still builds fine.
Next to that, a good CI, runs all the unit tests on each commit. This however requires that the unit tests are built.
A simple approach is, build the entire codebase, and next run the unit tests. Typically this is building way to much code, which is not needed in order to get the unit test executables.
Which raises the question, how do you determine in all those cmake targets, which executable targets are unit tests. Many of us, have solved this with scripting, assuming by convention that unit tests live in a sub-directory called tests, for example. And then parsed the CMakeLists.txt files in those directories, to gather the names of the targets, and made all those targets a dependency for so dummy “main” executable, which was ‘the thing to build’.
Whatever scripted solution, smart or stupid, finally a better approach has arrived with CMake 4.4.0. There is a downside, it does not support makefile based build systems, only Ninja. Once more a reason it seems to ditch makefiles, since also CMake support for modules is not supporting makefiles but it does support Ninja.
Who is still using Makefiles ? Who switched from Makefiles to Ninja, did you encounter issues ? Feel free to comment.
An artificial example
Imagine we have a silly codebase like the following:
It consists out of 4 executables, 3 of them are unit tests, and one is not (aka an executable or library which has no unit tests, and no other unit test depends on it, aka do not build this one when the unit tests needs to be build).
For the (unit) tests, the CMakeLists.txt are like this:
Very trivial indeed. What does cmake/ctests learn from this:
- we have an executable target (with it’s dependencies)
- a test is added, whom runs that executable target
So basically all information is present, if I would want to run that test, that it should first build that executable. Till this day (I think), nothing much happened with this information.
The top CMakeLists.txt looks like:
The line of importance is:
set(CMAKE_TEST_BUILD_DEPENDS ON)What happens when that line is not present yet, and we generated the build system like:
cmake -G Ninja -D CMAKE_BUILD_TYPE=Release -S . -B build/Releaseand running ctest:
ctest --test-dir=build/Release --parallel 16 --output-on-failureWe get a lot of errors, with the core problem being:
The following tests FAILED:
1 - Test1 (Not Run)
2 - Test2 (Not Run)
3 - Test3 (Not Run)because it could not find the test executables.
As described above, we need to ensure we first build those test executables, and preferably nothing more.
Enter the new feature
When we add the important line mentioned above:
set(CMAKE_TEST_BUILD_DEPENDS ON)we will get extra targets, as described here in the CMake 4.4.0 release notes (https://cmake.org/cmake/help/v4.4/release/4.4.html)
We get new targets, in our example, these are:
- test_prep/Test1
- test_prep/Test2
- test_prep/Test3
- test_prep/all
Now that latter one, is the answer to our prayers. Let’s try it out.
We invoke cmake as before. And we could build Test1 now in 2 ways:
- cmake –build ./build/Release/ –parallel 16 –target Test1
- cmake –build ./build/Release/ –parallel 16 –target test_prep/Test1
But we want the real deal:
cmake --build ./build/Release/ --parallel 16 --target test_prep/allWhen we invoke this, we can see (well I can overhere) the 3 tests are being build, and it’s dependencies, which is in my silly test setup catch2, and nothing more.
And now I can invoke ctest (see the outcome below, a test is failing, because it are educational tests to demonstrate ‘code issues’).
Test project /home/xxxxxxxxxxx/ctest/build/Release
Start 1: Test1
Start 2: Test2
Start 3: Test3
1/3 Test #1: Test1 ............................ Passed 0.00 sec
2/3 Test #2: Test2 ............................***Failed 0.00 sec
...
3/3 Test #3: Test3 ............................ Passed 0.00 sec
67% tests passed, 1 tests failed out of 3
Total Test time (real) = 0.01 sec
The following tests FAILED:
2 - Test2 (Failed)Conclusion
If you want to run a CI job, building (and building only) your unit tests, and run them, this is the recipe:
- at the top CMakeLists.txt have: set(CMAKE_TEST_BUILD_DEPENDS ON)
- generate with ninja generator your build system: cmake -G Ninja -D CMAKE_BUILD_TYPE=Release -S . -B build/Release
- build: cmake –build ./build/Release/ –parallel 16 –target test_prep/all
- test: ctest –test-dir=build/Release –parallel 16 –output-on-failure
Leave a Reply