Introducing a fast and small unit testing framework for C/C++ inspiried by GOLANG.

Repository and code

Introduction

Note: This applies to MacOS and Mach-O 64bit binaries.

I have not been a fan of the TDD approach in general. There are however cases where I do like it. Recently I’be been writing code for small/embedded systems (ESP8266/ESP32/ATmega) where the whole cycle of code/deploy/run/fix is a bit cumbersome. The deploy is pretty slow as compared to your average desktop. Debugging is more or less absent. In order to speed up my development and also to minimize the amount of time spent debugging I wanted to have a fast mechanism for running tests to verify functionality.

I am a fan of golang and the simple but effective approach golang takes on development. And I wanted some kind of minimalistic unit test framework that feels and works similar to golang’s unit testing (see: https://golang.org/pkg/testing/).

There are some key aspects of the golang test approach:

• Test code is kept close to the code in test
• Fast execution
• Minimal/No dependencies
• External runner (easily integrate testing as part of the build process)

Requirements

There are several unit testing frameworks (mainly C++) but generally they are big frameworks and brings in a lot of dependencies (see: https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks).

Instead I (as any other developer with self respect) cooked up my own based on the principles of golang unit testing.

• Allow test-code to be kept close with the code in test
• Require no or just a single header file
• Modular testing (as projects grow you want to split tests up in modules)
• Fast execution (test engine should minimize overhead)\
• Simple and clear output
• External test runner - independent of the code I am writing

In order to have an external test runner I knew I was needing to load binary code and to somehow find exported functions which would be a test function. I didn’t want the test-runner to depend on any other declaration (like a function list) in order to determine what to execute. At the end I determined that exported test-functions must reside in an dynamic library in order for the test runner to work. This was not what I wanted but as I am using CMAKE to drive my build process it is a minor problem - very easy to add a new build target with CMake.

Defining test functions and modules

The test runner must know what to execute. As C/C++ has no way to decorate functions with meta data the only option is to use name mangling. Remember, I don’t want to feed the test runner with an additional list of functions!

Test functions are defined like: int test_module_case(ITesting *t)

The test runner looks for any function matching this pattern. There are three categories of functions.

1. test_main, similar to application entry point but for test execution.
2. test_module, without test-case, entry point for modules
3. test_module_case, single test case

Order of execution:

• test_main is executed first
• test_module is executed before any cases for the module

Order of modules can be controlled when executing the test runner. Order of test cases can not be controlled.

All test functions takes a single argument (ITesting *t) and returns a test result code. The test result codes are defined like:

// Return codes from test functions
#define kTR_Pass 0x00
#define kTR_Fail 0x10
#define kTR_FailModule 0x20
#define kTR_FailAll 0x30

• kTR_Pass, pass current test, continue execution
• kTR_Fail, fail current test, continue execution
• kTR_FailModule, fail current test, skip rest of module
• kTR_FailAll, fail current test and stop testing altogether

The ITesting interface contains a few convenient functions - they can be skipped all together.

struct ITesting {
    // Just info output - doesn't affect test execution
    void (*Debug)(int line, const char *file, const char *format, ...); 
    void (*Info)(int line, const char *file, const char *format, ...); 
    void (*Warning)(int line, const char *file, const char *format, ...); 
    // Errors - affect test execution
    // Current test, proceed to next
    void (*Error)(int line, const char *file, const char *format, ...); 
    // Current test, stop module and proceed to next
    void (*Fatal)(int line, const char *file, const char *format, ...); 
    // Current test, stop execution
    void (*Abort)(int line, const char *file, const char *format, ...); 
};

You may skip all of this - just remember that the return code affects the test runner statistics and test pass/fail indication.

Test main

The test main is executed first of all tests. int test_main(ITesting *t) This gives the opportunity to set up global’s, initialize system variables and similar. It is an optional function, you don’t need to declare it.

Test module/package main

A module test-case without the case name is considered the main function for the module. int test_module(ITesting *t) This will be executed before any test case for this specific module.

Test cases

Test cases are declared as: int test_module_case(ITesting *t)

Pass/Fail Handling

Test cases return pass/fail indication back to the test-runner. It is also possible to call a function in the ITesting interface to indicate pass/fail if you still want to proceed with execution.

Example

#include <testinterface.h>
extern "C" {
    int test_time_ctorempty(ITesting *t);
}
int test_time_ctorempty(ITesting *t) {
    auto tm = Time();

    std::string expected("00:00:0.0000");
    std::string out = tm.Format();
    if (out != expected) {
        printf("Format failure, got: %s, expected: %s\n", out.c_str(), expected.c_str());
        return kTR_Fail;
    }
    return kTR_Pass;
}

Test execution

The test runner executes in the following order

1. test_main is executed (if present)
2. test_module is executed (if present)
3. test_module_case is executed (if present)

Step 2 and 3 is repeated for every module declared.

Test Runner

Clone and Build

1. Clone the gitrep: git clone https://github.com/gnilk/testrunner.git
2. Enter cloned rep: cd testrunner
3. Build and Install: cmake .; make; make install

If you want a cleaner root directory, do like this:

1. same
2. same
3. Create build directory mkdir build
4. Enter build directory cd build
5. Build and Install: cmake ..; make; make install

Note: test runner is installed in /usr/local/bin by default and testinterface.h is installed in /usr/local/include.

You may skip the install step and copy the testinterface.h directly into your own project.

Running the test runner

From your project build output directory just run trun. This will scan all directories for dynamic libraries (.dylib). Each dynamic library will be opened and scanned for functions matching the test pattern.

Execution control

The following options are availble for execution control. Default input is current directory.

TestRunner v0.1 - C/C++ Unit Test Runner
Usage: trun [options] input
Options: 
  -v  Verbose, increase for more!
  -d  Dump configuration before starting
  -g  No globals, skip globals (default: off)
  -s  Silent, surpress messages from test cases (default: off)
  -r  Discard return from test case (default: off)
  -c  Continue on module failure (default: off)
  -C  Continue on total failure (default: off)
  -m <list> List of modules to test (default: '-' (all))

Input should be a list dylib's to be tested (or directory)

Most commonly used argument is -m in order to control which module to be tested.

Examples: trun -m parser Execute any test case in the parser module searching from current directory. Will also execute test_main.

trun -sm parser Same as above but will not output any log messages when coming through the ITesting interface functions.

trun -sgm parser Same as above but also skip globals (-g).

trun -m parser,encoder,decoder Execute any test case in for parser, encoder and decoder. In that exact order.