Why Test-Driven Data Analysis?

OK, everything you need to know about TeX has been explained—unless you happen to be fallible. If you don't plan to make any errors, don't bother to read this chapter.

— The TeXbook, Chapter 27, Recovery from Errors. Donald E. Knuth.¹

The concept of test-driven data analysis seeks to improve the answers to two sets of questions, which are defined with reference to an "analytical process".

The questions assume that you have used the analytical process at least once, with one or more specific collections of inputs, and that you are ready to use, share, deliver or simply believe the results.

The questions in the first group concern the implementation of your analytical process:

Implementation Questions

1. How confident are you that the outputs produced by the analytical process, with the input data you have used, are correct?

2. How confident are you that the outputs would be the same if the analytical process were repeated using the same input data?

3. Does your answer change if you repeat the process using different hardware, or after upgrading the operating system or other software?

4. Would the analytical process generate any warning or error if its results were different from when you first ran it and satisfied yourself with the results?

5. If the analytical process relies on any reference data, how confident are you that you would know if that reference data changed or became corrupted?

6. If the analytical process were run with different input data, how confident are you that the output would be correct on that data?

7. If corrupt or invalid input data were used, how confident are you that the process would detect this and raise an appropriate warning, error or failure?

8. Would someone else be able reliably to produce the same results as you from the same inputs, given detailed instructions and access?

9. Corollary: do such detailed instructions exist? If you were knocked down by the proverbial bus, how easily could someone else use the analytical process?

10. If someone developed an equivalent analytical process, and their results were different, how confident are you that yours would prove to be correct?

These questions are broadly similar to the questions addressed by test-driven development, set in the specific context of data analysis.

The questions in our second group are concerned with the meaning of the analysis, and a larger, more important sense of correctness:

** Interpretation Questions**

1. Is the input data² correct?³
2. Is your interpretation of the input data correct?
3. Are the algorithms you are applying to the data meaningful and appropriate?
4. Are the results plausible?
5. Is your interpretation of the results correct?
6. More generally, what are you missing?

These questions are less clear cut than the implementation questions, but are at least as important, and in some ways are more important. If the implementation questions are about producing the right answers, the interpretation questions are about asking the right questions, and understanding the answers.

Over the coming posts, we will seek to shape a coherent methodology and set of tools to help us provide better answers to both sets of questions—implementational and interpretational. If we succeed, the result should be something worthy of the name test-driven data analysis.