Just testing is not enough, you need to know how much of your code you are actually testing to obtain any sort of confidence. In this article I show you my approach to combining the built-in
trace modules to perform unit testing and coverage testing in one go, without introducing external dependencies.
Coverage tests in Python
If you want to check how much of your code is covered by your tests you can of course use any number of available modules (coverage.py is an excellent starting point) but this does introduce a dependency on external code, something you might want to avoid.
Checking coverage however, is just as simple as building unit test with Python's bundled modules. In the example below we use
unittest to write our unit tests and use the less known
trace module to tally any lines executed when we execute our test runner.
Example unit test
The code to test is a rather silly example of a function that has three branches. The bulk of the code is about the unit tests and the final (from line 28 onward) shows how we can execute the test runner (
unittest.main()) within the
def f(a): if a>0: return a elif a==0: raise ValueError() return -a # end of coverage if __name__ == "__main__": import unittest import trace from os.path import dirname,abspath from collections import defaultdict from re import compile,match eoc=compile(r'^\s*# end of coverage') class TestFunction(unittest.TestCase): def test_int(self): self.assertEqual(3,f(3)) self.assertEqual(-3,f(-3)) def test_float(self): self.assertEqual(3.0,f(3.0)) self.assertEqual(-3.0,f(-3.0)) t=trace.Trace(count=1,trace=0) t.runfunc(unittest.main,exit=False) r=t.results() linecount = defaultdict(int) for line in r.counts: if line==__file__: linecount[line]=r.counts[line] with open(__file__) as f: for linenumber,line in enumerate(f,start=1): if eoc.match(line) : break print("%02d %s"%(linecount[linenumber],line),end='')
Using the trace module
We first create an instance of a
Trace object initialized with a
trace parameter set to zero to prevent printing every line it encounters. (Full documentation is available on the Python site.) Then we invoke the
runfunc() method with
unittest.main as its first parameter. This will trace the test runner and collect the results in the
Trace instance. The extra
exit=False is passed to the test runner and will prevent it from stopping the program once the tests are done (after all, we still want to do something with the results from the trace).
The next step is to retrieve the results (line 30). These results are available as a dictionary which keys are (filename,linenumber) tuples and the corresponding values the number of times this line is executed. We convert this to a default dictionary for those lines that are present in the current source file (line 34). This dictionary is indexed by line numbers.
The final step is to iterate over the lines in the current source file until we encounter a line that signal the end of the tested code (line 39). We print those lines with the number of times they were executes. The result may look like this
.. ---------------------------------------- Ran 2 tests in 0.031s OK 00 def f(a): 04 if a>0: 02 return a 02 elif a==0: 00 raise ValueError() 02 return -a
The tests ran ok but as is immediately obvious from line 10 is that we missed an important branch that is not covered by our unit tests. Note that the line with the
def statement is also not executed by our tests but this is to be expected as function definitions are only executed when the function is compiled, not when it is run.
RemarksWe do use more than just the document API of the
tracemodule because the
tracemodule as published has a major drawback: it not only collects statistics but only offers a public API that immediately prints those results. This not convenient when you want to postprocess the results or present them in a different way. We are therefore more or less forced to access the underlying data (the
countsattribute in the object returned by the