Lecture 5: Testing

\( \newcommand\bigO{\mathrm{O}} \)

1 Complete Test

  • Testing every possible path in every possible situation. For example, we can test the following function on all char inputs, there are only 256 characters after all:

    // If `c` is a letter, returns the corresponding upper-case letter. Otherwise,
    // returns `c`.
    char ToUpper(char c) {
      if ('a' <= c && c <= 'z') {
        return c - 'a' + 'A';
      return c;
    // Returns if the input is an alphabetical symbol
    bool IsAlpha(char c) {
      return 'A' <= ToUpper(c) && ToUpper(c) <= 'Z';
  • Complete tests are infeasible in general! For example, testing a distance function computing \( (x_1 - x_2)^2 + (y_1 - y_2)^2 \) would require \( 2^{32 \times 4} = 2^{128} \approx 10^{38} \) cases. That is a hundred trillion trillion trillion cases.
  • The best we can hope for is trying to approximate a complete test by testing various types of cases.
  • Another alternative is to "simulate" running all cases. This is in the domain of program analysis. You may learn about this in the future when you take Compilers or Programming Languages.
  • The more rigorous testing a program can "pass", the more confidence is gained that the program is "bulletproof" (handles every error as expected).

2 Unit Testing

  • Testing individual pieces (units) of a program (small and large) to ensure correct behavior.
  • Good software practice is structuring your program into modular units.
  • Good tests generally cover interesting cases including:
    • Normal Cases: Cases where the input is generally what we expect.
    • Boundary Cases: Cases that test the edge values of possible inputs.
    • Error Cases: Invalid cases (like passing a string instead of an int).
      • C++ catches most of these types of errors during the compilation process.
      • Other languages, such as Python, may need to rigorously check invalid input cases.

2.1 Property Testing

Sometimes a function has some properties. For example, if we have a maximum function max, then max(l) >= l[i] holds for all 0 <= i < l.size(). We can write tests that generate random inputs and test these properties if we want.

3 Test Suite

  • A program containing various tests confirming certain behavior.
  • A good way to automate tests.
    • Very important for large-scale projects where other engineers may make a change that affects functionality of other existing (or new) functionality.
  • We've seen testing programs in several of our lab assignments testing students' submissions throughout the quarter.

3.1 Metrics for Testing

Coverage is a usually common metric to see how well a test suite functions. The most common versions of it are:

Line coverage
How many lines of the code are run in tests
Branch coverage
How many different branches are taken

4 Example

Writing our own simple program using tddFuncs, which tests a function that takes four integers and returns the largest.

#include <iostream>
#include <string>
#include "tddFuncs.h"

using namespace std;

int biggest (int a, int b, int c, int d) {
    int biggest = 0;
    if (a >= b && a >= c && a >= d)
        return a;
    if (b >= a && b >= c && b >= d)
        return b;
    if (c >= a && c >= b && c >= d)
        return c;
    return d;

int isPositive(int a) {
    return a >= 0;

int main() {
    ASSERT_EQUALS(4, biggest(1,2,3,4));
    ASSERT_EQUALS(4, biggest(1,2,4,3));
    ASSERT_EQUALS(4, biggest(1,4,2,3));
    ASSERT_EQUALS(4, biggest(4,1,2,3));

    ASSERT_EQUALS(4, biggest(4,4,4,4));
    ASSERT_EQUALS(-1, biggest(-1,-2,-3,-4));
    ASSERT_EQUALS(0, biggest(-1,0,-3,-4));


    return 0;

5 Test-Driven Development

  • Write test cases that describe what the intended behavior of a unit of software should BEFORE implementing the functionality.
    • Defines the requirements of your piece of software.
  • Implement the details of the functionality with the intention of passing the tests.
  • Repeat until the tests pass.
  • Imagine large software products where dozens of engineers are trying to add new features / implement optimizations all at the same time.
    • Having a "suite" of tests before deploying software to the public is essential.
    • Someone may modify changes that work for a current version, but breaks functionality in another version
    • Rigorous tests enable confidence in the stability in software.

5.1 Mocking

Sometimes our code depends on some external classes or functions (such as file I/O, databases, connecting to network). To test a piece of code in isolation, we create mock versions of those classes and functions that work for the test inputs. The mock classes allow us to reduce the parts of code we need to look at when a test fails.

Author: Mehmet Emre


The material for this class is based on Prof. Richert Wang's material for CS 32