Reference Testing Exercise 2 (pytest flavour)

Posted on Thu 31 October 2019 in TDDA • Tagged with reference test, exercise, screencast, video, pytest

This exercise (video 2m 58s) shows a powerful way to run only a single test, or some subset of tests, by using the @tag decorator available in the TDDA library. This is useful for speeding up the test cycle and allowing you to focus on a single test, or a few tests. We will also see, in the next exercise, how it can be used to update test results more easily and safely when expected behaviour changes.

(If you do not currently use pytest for writing tests, you might prefer the unittest-flavoured version of this exercise, since unittest is in Python's standard library.)

Prerequisites

★ You need to have the TDDA Python library (version 1.0.31 or newer) installed see installation. Use

tdda version

to check the version that you have.

Step 1: Copy the exercises (if you don't already have them)

You need to change to some directory in which you're happy to create three directories with data. We are use ~/tmp for this. Then copy the example code.

$ cd ~/tmp
$ tdda examples    # copy the example code

Step 2: Go the exercise files and examine them:

$ cd referencetest_examples/exercises-pytest/exercise2  # Go to exercise2

As in the first exercise, you should have at least the following four files:

$ ls
conftest.py expected.html   generators.py   test_all.py
  • conftest.html is configuration to extend pytest with referencetest capabilities,
  • expected.html contains the expected output from one test,
  • generators.py contains the code to be tested,
  • test_all.py contains the tests.

If you look at test_all.py, you'll see it contains five test functions. Only one of the tests is useful (testExampleStringGeneration) with all the others making manifestly true assertions and most of them deliberately wasting time to simulate annoyingly slow tests.

from generators import generate_string

def testZero():
    assert True

def testOne():
    time.sleep(1)
    assert 1 == 1

def testExampleStringGeneration(ref):
    actual = generate_string()
    ref.assertStringCorrect(actual, 'expected.html')

def testTwo():
    time.sleep(2)
    assert 2 == 2

def testThree():
    time.sleep(3)
    assert 3 == 3

Step 3: Run the tests, which should be slow and produce one failure

$ pytest           #  This will work with Python 3 or Python2

When you run the tests, you should get a single failure, that being the non-trivial test testExampleStringGeneration.

The output will be something like:

============================= test session starts ==============================
test_all.py ..F..

[...details of test failure...]

====================== 1 failed, 4 passed in 6.17 seconds ======================

We get a test failure because we haven't added the ignore_substrings parameter that we saw in Exercise 1 is needed for it to pass.

The tests should take slightly over 6 seconds in total to run, because of the three annoyingly slow tests with sleep statements in them—testOne, testTwo and testThree. (If you're not annoyed by a 6-second delay, increase the sleep time in one of the "sleepy" tests until you are annoyed!)

The point of this exercise is to show some simple but very useful functionality for running only tests on which we wish to focus, such as our failing test.

Step 4: Tag the failing test using @tag

The TDDA library includes a function called tag; this is a decorator function1 that we can put before individual tests, to mark them as being of special interest temporarily.

Edit test_all.py to decorate the failing test by an import statement to bring in tag from the TDDA library, and then decorate the definition of testStringFunction by preceding it with @tag as follows:

from tdda.referencetest import tag

def testZero():
    assert True

def testOne():
    time.sleep(1)
    assert 1 == 1

@tag
def testExampleStringGeneration(ref):
    actual = generate_string()
    ref.assertStringCorrect(actual, 'expected.html')

Step 5: Run only the tagged test

Having tagged the failing test, if we run the tests again adding --tagged to the command, it will run only the tagged test, and take hardly any time. The (abbreviated) output should be something like

============================= test session starts ==============================
$ pytest --tagged
test_all.py F

[...details of test failure...]

=========================== 1 failed in 0.16 seconds ===========================

We can tag as many tests as we like, across any number of test files, to run a subset of tests, rather than a single one.

Step 6: Locating @tag decorators

In a typical debugging or test development cycle in which you have been using the @tag decorator to focus on just a few failing tests, you might end up with @tag decorations scattered across several files, perhaps in multiple directories.

Although it's not hard to use grep or grep -r to find them, the library can actually do this for you. If you use the --istagged flag instead of running the tests, the library will report which test classes in which files have tagged tests. So in our case:

$ pytest --istagged
============================= test session starts ==============================
platform darwin -- Python 3.7.3, pytest-4.4.0, py-1.8.0, pluggy-0.9.0
rootdir: /Users/njr/tmp/referencetest_examples/exercises-pytest/exercise2
collecting ...
test_all.testExampleStringGeneration
collected 5 items

========================= no tests ran in 0.01 seconds =========================

Obviously, in the case of a single test file, this is not a big deal, but if you have dozens or hundreds of source files, in a directory hierarchy, and have tagged a few functions across them, it becomes significantly more helpful.

Recap: What we have seen

This simple exercise has shown how we can easily run subsets of tests by tagging them and then using --tagged to run only tagged tests.

In this case, the motivation was simply to save time and reduce clutter in the output, focusing on one test, or a small number of tests.

In the Exercise 3, we will see how this combines with the ability to automatically regenerate updated reference outputs to make for a safe and efficient way to update tests after code changes.


  1. Decorator functions in Python are functions that are used to transform other functions: they take a function as an argument and return a new function that modifies the original in some way. Out decorator function tag is called by writing @tag on the line before function (or class) definition, and the effect of this is that the function returned by @tag replaces the function (or class) it precedes. In our case, all @tag does is set an attribute on the function in question so that the TDDA reference test framework can identify it as a tagged function, and choose to run only tagged tests when so requested. 


Reference Testing Exercise 2 (unittest flavour)

Posted on Wed 30 October 2019 in TDDA • Tagged with reference test, exercise, screencast, video, unittest

This exercise (video 3m 34s) shows a powerful way to run only a single test, or some subset of tests, by using the @tag decorator available in the TDDA library. This is useful for speeding up the test cycle and allowing you to focus on a single test, or a few tests. We will also see, in the next exercise, how it can be used to update test results more easily and safely when expected behaviour changes.

(If you use pytest for writing tests, you might prefer the pytest-flavoured version of this exercise.)

Prerequisites

★ You need to have the TDDA Python library (version 1.0.31 or newer) installed see installation. Use

tdda version

to check the version that you have.

Step 1: Copy the exercises (if you don't already have them)

You need to change to some directory in which you're happy to create three directories with data. We are use ~/tmp for this. Then copy the example code.

$ cd ~/tmp
$ tdda examples    # copy the example code

Step 2: Go the exercise files and examine them:

$ cd referencetest_examples/exercises-unittest/exercise2  # Go to exercise2

As in the first exercise, you should have at least the following three files

$ ls
expected.html   generators.py   test_all.py
  • expected.html contains the expected output from one test,
  • generators.py contains the code to be tested,
  • test_all.py contains the tests.

If you look at test_all.py, you'll see it contains two test classes with five tests between them. Only one of the tests is useful (testExampleStringGeneration) with all the others making manifestly true assertions and deliberately wasting time to simulate annoyingly slow tests.

import time
from tdda.referencetest import ReferenceTestCase, tag
from generators import generate_string

class TestQuickThings(ReferenceTestCase):

    def testExampleStringGeneration(self):
        actual = generate_string()
        self.assertStringCorrect(actual, 'expected.html')

    def testZero(self):
        self.assertIsNone(None)


class TestSuperSlowThings(ReferenceTestCase):

    def testOne(self):
        time.sleep(1)
        self.assertEqual(1, 1)

    def testTwo(self):
        time.sleep(2)
        self.assertEqual(2, 2)

    def testThree(self):
        time.sleep(3)
        self.assertEqual(3, 3)

Step 3: Run the tests, which should be slow and produce one failure

$ python test_all.py   #  This will work with Python 3 or Python2

When you run the tests, you should get a single failure, that being the non-trivial test testExampleStringGeneration from the class TestQuickThings.

The output will be:

F....

[...details of test failure...]

Ran 5 tests in 6.007s
FAILED (failures=1)

We get a test failure because we haven't added the ignore_substrings parameter that we saw in Exercise 1 is needed for it to pass.

The tests should take slightly over 6 seconds in total to run, because of the annoyingly slow tests in TestSuperSlowThings. (If you're not annoyed by a 6-second delay, increase the sleep time in one of the "slow" tests until you are annoyed!)

The point of this exercise is to show some simple but very useful functionality for running only tests on which we wish to focus, such as our failing test.

Step 4: Tag the failing test using @tag

If you look at the import statements, you'll see that as well as ReferenceTestCase we also import tag. This is a decorator function1 that we can put before individual tests, or test classes, to indicate that they are of special interest temporarily.

Edit test_all.py to decorate the failing test by adding @tag on the line before it, thus:

class TestQuickThings(ReferenceTestCase):

    @tag
    def testExampleStringGeneration(self):
        actual = generate_string()
        self.assertStringCorrect(actual, 'expected.html')

    def testZero(self):
        self.assertIsNone(None)

Step 5: Run only the tagged test

Having tagged the failing test, if we run the tests again adding -1 (the digit one, for "single",not the letter ell) to the command, it will run only the tagged test, and take hardly any time. The (abbreviated) output should be something like

$ python test_all.py -1
F

[...details of test failure...]

Ran 1 tests in 0.006s
FAILED (failures=1)

You can also use --tagged instead of -1 if you like more descriptive flags.

We can tag as many tests as we like, across any number of test files, and we can also tag whole classes by placing the @tag decorator before a test class definition. So if we instead use:

@tag
class TestQuickThings(ReferenceTestCase):

    def testExampleStringGeneration(self):
        actual = generate_string()
        self.assertStringCorrect(actual, 'expected.html')

    def testZero(self):
        self.assertIsNone(None)

and run the tests with -1, we will get output more like:

$ python test_all.py -1
F.

[...details of test failure...]

Ran 2 tests in 0.006s
FAILED (failures=1)

In this case, both the tests in our first test class were run, but no others (and, in particular, not our painfully slow tests!)

Step 6: Locating @tag decorators

In a typical debugging or test development cycle in which you have been using the @tag decorator to focus on just a few failing tests, you might end up with @tag decorations scattered across several files, perhaps in multiple directories. (We're assuming here you have test_all.py or similar that imports all the other test classes so you can easily run them all together.)

Although it's not hard to use grep or grep -r to find them, the library can actually do this for you. If you use the -0 flag (the digit zero, for "no tests"), or the --istagged flag, instead of running the tests, the library will report which test classes in which files have tagged tests. So in our case:

$ python test_all.py -0

produces:

__main__.TestQuickThings

Here, __main__ stands for the current file; other files would be referenced by their imported name.

Recap: What we have seen

This simple exercise has shown how we can easily run subsets of tests by tagging them and then using the -1 flag (or --tagged) to run only tagged tests.

In this case, the motivation was simply to save time and reduce clutter in the output, focusing on one test, or a small number of tests.

In the Exercise 3, we will see how this combines with the ability to automatically regenerate updated reference outputs to make for a safe and efficient way to update tests after code changes.


  1. Decorator functions in Python are functions that are used to transform other functions: they take a function as an argument and return a new function that modifies the original in some way. Out decorator function tag is called by writing @tag on the line before function (or class) definition, and the effect of this is that the function returned by @tag replaces the function (or class) it precedes. In our case, all @tag does is set an attribute on the function in question so that the TDDA reference test framework can identify it as a tagged function, and choose to run only tagged tests when so requested. 


Reference Testing Exercise 1 (pytest flavour)

Posted on Tue 29 October 2019 in TDDA • Tagged with reference test, exercise, screencast, video, pytest

This exercise (video 8m 54s) shows how to migrate a test from using pytest directly to the exploiting the referencetest capabilities in the TDDA library. (If you do not currently use pytest for writing tests, you might prefer the unittest-flavoured version of this exercise, since unittest is in Python's standard library.)

We will see how even simple use of referencetest

  • makes it much easier to see how tests have failed when complex outputs are generated
  • helps us to update reference outputs (the expected values) when we have verified that a new behaviour is correct
  • allows us easily to write tests of code whose outputs are not identical from run to run. We do this by specifying exclusions from the comparisons used in assertions.

Prerequisites

★ You need to have the TDDA Python library installed (version 1.0.31 or newer) see installation. Use

tdda version

to check the version that you have.

Step 1: Copy the exercises

You need to change to some directory in which you're happy to create three new directories with data. We are use ~/tmp for this. Then copy the example code.

$ cd ~/tmp
$ tdda examples    # copy the example code

Step 2: Go the exercise files and examine them:

$ cd referencetest_examples/exercises-pytest/exercise1  # Go to exercise1

You should have at least the following four files:

$ ls
conftest.py expected.html   generators.py   test_all.py
  • generators.py contains a function called generate_string that, when called, returns HTML text suitable for viewing as a web page.

  • expected.html is the result of calling that function, saved to file

  • test_all.py contains a single unittest-based test of that file.

  • conftest.py imports key referencetest functionality from the tdda library into pytest.

It's probably useful to look at the web page expected.html in a browser, either by navigating to it in a file browser and double clicking it, or by using

open expected.html

if your OS supports this. As you can see, it's just some text and an image. The image is an inline SVG vector image, generated along with the text.

Also have a look at the test code. The core part of it is very short:

from generators import generate_string

def testExampleStringGeneration():
    actual = generate_string()
    with open('expected.html') as f:
        expected = f.read()
    assert actual == expected

The code

  • calls generate_string() to create the content
  • stores its output in the variable actual
  • reads the expected content into the variable expected
  • asserts that the two strings are the same.

Step 3. Run the test, which should fail

$ pytest      #  This will whether pytest uses Python2 or Python3

You should get a failure, and pytest tries quite hard to show what's causing the failure:

=================================== FAILURES ===================================
_________________________ testExampleStringGeneration __________________________

    def testExampleStringGeneration():
        actual = generate_string()
        with open('expected.html') as f:
            expected = f.read()
>       assert actual == expected
E       AssertionError: assert '<!DOCTYPE ht...y>\n</html>\n' == '<!DOCTYPE htm...y>\n</html>\n'
E         Skipping 69 identical leading characters in diff, use -v to show
E         -  Solutions, 2016
E         +  Solutions Limited, 2016
E         ?           ++++++++
E         -     Version 1.0.0
E         ?             ^
E         +     Version 0.0.0...
E
E         ...Full output truncated (31 lines hidden), use '-vv' to show

test_all.py:24: AssertionError
=========================== 1 failed in 0.11 seconds ===========================

You can certainly see that there's a different in the Version number in the output and also a line including 2016 (a copyright notice, in fact).

But it also says:

...Full output truncated (31 lines hidden), use '-vv' to show

and if you do that, the output becomes a bit overwhelming.

We'll convert the test to use the TDDA libraries referencetest and see how that helps.

Step 4. Change the code to use referencetest.

The key change we need to make is the to the assertion, which will now be:

  • ref.assertStringCorrect(actual, 'expected.html')

ref is object made available by conftest.py, and is passed into our test function by pytest. We therefore need to change the function declaration to take ref as an argument:

  • def testExampleStringGeneration(ref):

Finally, because assertStringCorrect compares a string in memory to content from a file, we don't need the lines in the middle that read the file:

* Delete the middle two lines of the test function.

The result is:

from generators import generate_string

def testExampleStringGeneration(ref):
    actual = generate_string()
    ref.assertStringCorrect(actual, 'expected.html')

Step 5. Run the modified test

$ pytest

You should see very different output, that includes, near the end, something like this:

E       AssertionError: 2 lines are different, starting at line 5
E       Expected file expected.html
E       Compare raw with:
E           diff /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-raw-expected.html expected.html
E
E       Compare post-processed with:
E           diff /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-expected.html /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/expected-expected.html

/Users/njr/python/tdda/tdda/referencetest/referencepytest.py:187: AssertionError

(You will probably need to scroll right to see all of the message on this page.)

Because the test failed, the TDDA library has written a copy of the actual ouput to file to make it easy for us to examine it and to use diff commands to see how it actually differs from what we expected. (In fact, it's written out two copies, a "raw" and a "post-precocessed" one, but we haven't used any processing, so they will be the same in our case. So we ignore the second diff command suggested for now.)

It's also given us the precise diff command we need to see the differences between our actual and expected output.

Step 6. Copy the first diff command and run it. You should see something similar to this:

$ diff /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-raw-expected.html expected.html
5,6c5,6
<     Copyright (c) Stochastic Solutions, 2016
<     Version 1.0.0
—
>     Copyright (c) Stochastic Solutions Limited, 2016
>     Version 0.0.0
35c35
< </html>
\ No newline at end of file
—
> </html>

(If you have a visual diff tool, can also use that. For example, on a Mac, if you have Xcode installed, you should have the opendiff command available.)

The diff makes it clear that there are three differences:

  • The copyright notice has changed slightly
  • The version number has changed
  • The string doesn't have a newline at the end, whereas the file does.

The Copyright and version numbers lines are both in comments in the HTML, so don't affect the rendering at all. You might want to confirm that if you look at the actual file it saved (/var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-raw-expected.html, the first file in the diff command), you should see that it looks identical.

In this case, therefore, we might now feel that we should simply update expected.html with what generate_string() is now producing. It would be (by design) extremely easy to change the diff in the command it gave is to cp to achieve that.

However, there's better thing we can do in this case.

Step 7. Specify exclusions

Standing back, it seems obvious likely that periodically the version number and Copyright line written to comments in the HTML will change. If the only difference between out expected output and what we actually generate are those, we'd probably prefer the test didn't fail.

The ref.assertStringCorrect function from referencetest gives us several mechanisms for specifying changes that can be ignored when checking whether a string is correct. The simplest one, which will be enough for our example, is just to specify strings which, if they occur on a line in the output, case differences in those lines to be ignored, so that the assertion doesn't fail.

Step 7a. Add the ignore_substrings parameter to assertStringCorrect as follows:

        ref.assertStringCorrect(actual, 'expected.html',
                                ignore_substrings=['Copyright', 'Version'])

Step 7b. Run the test again. It should now pass:

$ pytest
============================= test session starts ==============================

test_all.py .                                                            [100%]

=========================== 1 passed in 0.04 seconds ===========================

Recap: What we have seen

We've seen

  1. Converting standard pytest-based tests to use referencetestcase is straightfoward.

  2. When we do that, we gain access to powerful new kinds of assertion such as assertStringCorrect. Among the immediate benefits:

    • When there is failure, this saves the failing output to a temporary file
    • It tells you the exact diff command you need to see be able to see differences
    • This also makes it very easy to copy the new "known good" answer into place if you've verified that the new answer is now correct. (In fact, the library also has a more powerful way to do this, as we'll see in a later exercise).
  3. The ref.assertStringCorrect fucntion also has a number of mechanisms for allowing specific expected differences to occur without causing the test to fail. The simplest of these mechanisms is the ignore_substrings keyword argument we used here.


Reference Testing Exercise 1 (unittest flavour)

Posted on Mon 28 October 2019 in TDDA • Tagged with reference test, exercise, screencast, video, unittest

This exercise (video 8m 53s) shows how to migrate a test from using unittest directly to the exploiting the referencetest capabilities in the TDDA library. (If you use pytest for writing tests, you might prefer the pytest-flavoured version of this exercise.)

We will see how even simple use of referencetest

  • makes it much easier to see how tests have failed when complex outputs are generated
  • helps us to update reference outputs (the expected values) when we have verified that a new behaviour is correct
  • allows us easily to write tests of code whose outputs are not identical from run to run. We do this by specifying exclusions from the comparisons used in assertions.

Prerequisites

★ You need to have the TDDA Python library (version 1.0.31 or newer) installed see installation. Use

tdda version

to check the version that you have.

Step 1: Copy the exercises

You need to change to some directory in which you're happy to create three new directories with data. We are use ~/tmp for this. Then copy the example code.

$ cd ~/tmp
$ tdda examples    # copy the example code

Step 2: Go the exercise files and examine them:

$ cd referencetest_examples/exercises-unittest/exercise1  # Go to exercise1

You should have at least the following three files:

$ ls
expected.html   generators.py   test_all.py
  • generators.py contains a function called generate_string that, when called, returns HTML text suitable for viewing as a web page.

  • expected.html is the result of calling that function, saved to file

  • test_all.py contains a single unittest-based test of that file.

It's probably useful to look at the web page expected.html in a browser, either by navigating to it in a file browser and double clicking it, or by using

open expected.html

if your OS supports this. As you can see, it's just some text and an image. The image is an inline SVG vector image, generated along with the text.

Also have a look at the test code. The core part of it is very short:

import unittest

from generators import generate_string

class TestFileGeneration(unittest.TestCase):
    def testExampleStringGeneration(self):
        actual = generate_string()
        with open('expected.html') as f:
            expected = f.read()
        self.assertEqual(actual, expected)

if __name__ == '__main__':
    unittest.main()

The code

  • calls generate_string() to create the content
  • stores its output in the variable actual
  • reads the expected content into the variable expected
  • asserts that the two strings are the same.

Step 3. Run the test, which should fail

$ python test_all.py   #  This will work with Python 3 or Python2

You should get a failure, but it will probably be quite hard to see exactly what the differences are.

We'll convert the test to use the TDDA libraries referencetest and see how that helps.

Step 4. Change the code to use referencetest.

First we need our test to use ReferenceTestCase from tdda.referencetest instead of unittest.TestCase. ReferenceTestCase is a subclass of unittest.TestCase.

  • Change the import statement to from tdda.referencetest import ReferenceTestCase
  • Replace unittest.TestCase with ReferenceTestCase in the class declaration
  • Replace unittest.main() with ReferenceTestCase.main()

The result is:

from tdda.referencetest import ReferenceTestCase

from generators import generate_string

class TestFileGeneration(ReferenceTestCase):
    def testExampleStringGeneration(self):
        actual = generate_string()
        with open('expected.html') as f:
            expected = f.read()
        self.assertEqual(actual, expected)

if __name__ == '__main__':
    ReferenceTestCase.main()

If you run this, it's behaviour should be exactly the same, because we haven't used any of the extra features of tdda.referencetest yet.

Step 5. Change the assertion to use assertStringCorrect

TDDA's ReferenceTestCase provides the assertStringCorrect method, which expects as its first positional arguments an actual string and the path to a file containing the expected result. So:

  • Change assertEqual to assertStringCorrect
  • Change expected to expected.html as the second argument to the assertion
  • Delete the two lines reading the file and assigning to expected as we no longer need that.
    def testExampleStringGeneration(self):
        actual = generate_string()
        self.assertStringCorrect(actual, 'expected.html')
    

Step 6. Run the modified test

$ python test_all.py

You should see very different output, that includes, near the end, something like this:

Expected file expected.html
Compare raw with:
    diff /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-raw-expected.html expected.html

Compare post-processed with:
    diff /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-expected.html /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/expected-expected.html

Because the test failed, the TDDA library has written a copy of the actual ouput to file to make it easy for us to examine it and to use diff commands to see how it actually differs from what we expected. (In fact, it's written out two copies, a "raw" and a "post-precocessed" one, but we haven't used any processing, so they will be the same in our case. So we ignore the second diff command suggested for now.)

It's also given us the precise diff command we need to see the differences between our actual and expected output.

Step 6a. Copy the first diff command and run it. You should see something similar to this:

$ diff /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-raw-expected.html expected.html
5,6c5,6
<     Copyright (c) Stochastic Solutions, 2016
<     Version 1.0.0
—
>     Copyright (c) Stochastic Solutions Limited, 2016
>     Version 0.0.0
35c35
< </html>
\ No newline at end of file
—
> </html>

(If you have a visual diff tool, can also use that. For example, on a Mac, if you have Xcode installed, you should have the opendiff command available.)

The diff makes it clear that there are three differences:

  • The copyright notice has changed slightly
  • The version number has changed
  • The string doesn't have a newline at the end, whereas the file does.

The Copyright and version numbers lines are both in comments in the HTML, so don't affect the rendering at all. You might want to confirm that if you look at the actual file it saved (/var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-raw-expected.html, the first file in the diff command), you should see that it looks identical.

In this case, therefore, we might now feel that we should simply update expected.html with what generate_string() is now producing. It would be (by design) extremely easy to change the diff in the command it gave is to cp to achieve that.

However, there's better thing we can do in this case.

Step 7. Specify exclusions

Standing back, it seems obvious likely that periodically the version number and Copyright line written to comments in the HTML will change. If the only difference between out expected output and what we actually generate are those, we'd probably prefer the test didn't fail.

The assertStringCorrect method from referencetest gives us several mechanisms for specifying changes that can be ignored when checking whether a string is correct. The simplest one, which will be enough for our example, is just to specify strings which, if they occur on a line in the output, case differences in those lines to be ignored, so that the assertion doesn't fail.

Step 7a. Add the ignore_substrings parameter to assertStringCorrect as follows:

        self.assertStringCorrect(actual, 'expected.html',
                                 ignore_substrings=['Copyright', 'Version'])

Step 7b. Run the test again. It should now pass:

$ python3 test_all.py
.
----------------------------------------------------------------------
Ran 1 test in 0.002s

OK

Recap: What we have seen

We've seen

  1. Converting unittest-based tests to use ReferenceTestCase is straightfoward.

  2. When we do that, we gain access to powerful new assert methods such as assertStringCorrect. Among the immediate benefits:

    • When there is failure, this method saves the failing output to a temporary file
    • It tells you the exact diff command you need to see be able to see differences
    • This also makes it very easy to copy the new "known good" answer into place if you've verified that the new answer is now correct. (In fact, the library also has a more powerful way to do this, as we'll see in a later exercise).
  3. The assertStringCorrect method also has a number of mechanisms for allowing specific expected differences to occur without causing the test to fail. The simplest of these mechanisms is the ignore_substrings keyword argument we used here.


Screencasts and Exercises

Posted on Fri 25 October 2019 in TDDA • Tagged with tests, screencast, video, exercises

We've started producing a series of exercises for various aspects of TDDA, available on the blog, with follow-along screencasts.

There will be a series of posts about these, starting on Monday (28th October). There's a YouTube channel as well, if you want to subscribe.

The goal has been for each exercise to be as short and simple as it can reasonably be while still covering useful aspects.

The first set of exercises will cover the reference testing capabilities of TDDA, and at least some of them will be available both as unittest-favoured versions and pytest variants. If you don't currently use either, you probably want to follow the unittest variants, since unittest is part of Python's standard library.

There's a page for the exercises at:

tdda.info/exercises

which we'll try to keep up-to-date as we add more.

Please note: if you want to do the exercises, you'll need the latest TDDA release, and as we add more (unfortunately) you'll probably need to update each time we add new exercises, with something like

pip install -U tdda

or

python3 -m pip install -U tdda

depending on your setup. See the installation instructions for details.