Isolated tests

Looking through the rspec documentation, I find this...

We strongly recommend that you use RSpec’s mocking/stubbing framework to intercept class level calls like :find, :create and even :new to introduce mock instances instead of real active_record instances.

This allows you to focus your specs on the things that the controller does and not worry about complex validations and relationships that should be described in detail in the Model Examples

This notion that we should isolate every component and test it entirely by itself is something that I've heard over and over again. At one point, I even believed that it was a good idea. Such is the way of an anti-pattern. In the beginning, it usually appears to be a good idea and it's not until you've actually tried it that you start to realize that it might not have been so good.

A long time ago, I worked on a project where we followed this practice. Where the only time we touched the database was when we were testing the persistence layer. For each layer above that, we would mock out what came below so that each layer was tested in complete isolation. We had lots of tests and they all passed.

So when we showed the application to our client and it started blowing up, we were shocked. We had all kinds of tests! They all passed! How was it possible that anything was breaking?

We identified and fixed all those bugs and went back to show the client again. Once again, all our tests passed. Once again, the application blew up over and over again.

Looking at all the bugs that had been discovered, we noticed some interesting patterns. All of the bugs that our client was finding were in interactions BETWEEN layers. Each layer by itself was covered by tests and was working fine. The interactions between layers, however, were not being tested in any reasonable way and that's where all the bugs were.

I've seen this same idea in many applications. Whenever there is a strong emphasis on testing in isolation, many bugs are hidden until much later in the process. Sometimes not to be found until the application enters production.

The solution to this problem is simple. Don't ever assume that your isolated tests are enough. You always have to have tests that pass through all parts of the application.