TDD CANNOT Work

- Hemil Ruparel 25 June, 2024

Edit: This blog lacks formalism and is based on my assumptions. Refer to A formal analysis of Iterated TDD for a more complete idea. Rest of the article is present verbatim.

Please note: I am not saying it does not work sometimes. I am saying it fundamentally CANNOT work

Proof?

While, I have no hard data about why this is the case, the idea is this: (I have no formal proof) but I will try to (informally) convince you all, that TDD can be reduced to Hill Climbing algorithm. And if that is the case, then, it has some problems which you fundamentally cannot solve with Hill Climbing (TDD).

This effectively makes TDD unusable for an entire class of problems. Again, I have no hard data. It is a good question to ask what is the probability that given a random uniform distribution of problems, what is the probability that TDD will work for me? (for, whatever definition of work you pick).

Edit: I showed this to Nabarun Mondal. He told me, Hill Climbing can yet again be generalized to fixed point iteration. But I do not fully understand it, so I wont comment on it yet. I will try to figure it out and update this blog post

What is TDD?

I will go with the definition of Kent Beck as he "rediscovered" TDD. In the book, TDD by example, Kent explains:

Here's what we do: we drive development with automated tests, a style of development called Test-Driven Development (TDD). In Test-Driven Development, we

• Write new code only if an automated test has failed
• Eliminate duplication

Further down, he says:

The two rules imply an order to the tasks of programming.

1. Red - Write a little test that doesn't work, and perhaps doesn't even compile at first.
2. Green - Make the test work quickly, committing whatever sins necessary in the process.
3. Refactor - Eliminate all of the duplication created in merely getting the test to work.

Notice how TDD is a fundamentally iterative approach

Effectively, what we are doing is:

• Assess the current situation
• Make a small change
• Course correct yourself based on the feedback

This has a name - it is called Hill Climbing algorithm. From Wikipedia:

In numerical analysis, hill climbing is a mathematical optimization technique which belongs to the family of local search. It is an iterative algorithm that starts with an arbitrary solution to a problem, then attempts to find a better solution by making an incremental change to the solution. If the change produces a better solution, another incremental change is made to the new solution, and so on until no further improvements can be found.

Notice the similarities: you start with an arbitrary test, make an incremental change, if that incremental change is better (code or tests), then we keep on making the incremental change (new tests and refactored code) until no further improvements can be found (London TDDer is satisfied with the design and Chicago TDDer is satisfied with the regression suite).

There are lots of problems with this approach though:

• Local Maxima - If the heuristic is not convex, hill climbing may not find the optimal solution. Now, while I have no proof that the heuristic is not convex, my experience says it is. The sheer number of variables - teams, projects, motivations, skills, monetary benefit and inifinite others make me very hard to believe the heuristic is not convex (but again I may be wrong 😄)
• Ridges and valleys - Because you are making very tiny steps, in Red Greeen Refactor cycle 300, you may realise you made a mistake in cycle 3 and now fixing it, would require a lot of effort as a lot of code depends on it
• Plateau - Sometimes, you can think of x amount of things you can do to fix the code and they all seem equally good, because of which you can't figure out what to do and you may pick unoptimal solutions

All of this just to say, there are a lot of cases where TDD will just pick unoptimal solutions. Now the interesting question is, given an arbitrary problem, what is the probability that TDD will give us a good enough solution (you can define good enough to be whatever hueristic you want). Answer is I do not know. But what I do know, is this idea that you can develop any interesting software incrementally is absurd. You will most certainly make unoptimal choices and sometimes the cost of correcting the mistakes is the dreaded complete rewrite which again has no guarantee to fix your problems.

Now, TDD can "work" technically in that you may have working code. But I guess my definition of work is different. I want "good" solutions. Not "good enough" solutions. I believe TDD will be stuck in local maxima for almost any non trivial problem. And you will almost always end up with sub optimal solutions. If that is fine, then yes, TDD does work, but really does it?

What is the solution?

All I can say is understand the fundamental mathematical problem you are trying to solve. If you can't do it, either you are building CRUD apps or you dont know what you dont know. If you can do it, you do not need any design methodology. Testing is a completely different topic. Any good tester will tell you TDD is stupid way to build regression tests. And if you do believe otherwise, here is a problem:

We have implemented a function f(x) = sin(x). How do we test it? How do we design a solution for it?

Thank you to Nabarun Mondal for pushing me to engineering and science and not religion. I hope I have done justice to it.