What is “test-driven development for web applications”?
One day I would like to write a post called “test-driven development for web applications”. So apologies to anyone who came here looking for that. Because I have no idea what that is. But maybe you do? Please feel free to leave a comment.
I would also like to write a tutorial called “how to write unit tests for web applications” because almost every example I’ve come across (and I admit, I have not looked hard) of how to write unit tests tends to involve obvious classes like BankAccount or abstract classes like ObjectFactory. Sheesh.
At work I help lead the development of a web application written in PHP on top of a MySQL database. More than a year ago we decided: “we need unit tests”. So we took the typical approach and used PHPUnit to generate a stub for each of our core “model” classes. Starting with the shortest and easiest classes I began writing tests. What a pain in the ass. The one good thing that came out of it is that I made it possible to do a single command install of our codebase.
Mostly the tests tested whether inserting, updating, and selecting records out of the database and loading them into objects succeeded. Over and over and over again. And every time we had to add a field to a table, we had to update the test in several repetitive, tedious ways that ensured that our first foray into testing was bound to fail. I kept at it for a while. Most of the bugs I found were due to out of date tests. And you probably know how this story ends.
Skip ahead to last Monday, and Ivan informs me that Sebastian Bergmann, the creator of PHPUnit, is giving a talk over at CBS Interactive (nee CNet). I figure why not. Maybe I’d learn something. But when he got to this slide, I realized this talk was not going to get me any closer to the holy grail of “how to write unit tests for web applications”:
I couldn’t contain an outburst at that point, “But isn’t that like every application?” How had it come to this, me a belligerent heckler at a unit testing meetup?
It reminds me that Jonathan recently sent me a link to Tim Bray’s Test-Driven Heresy post with the email subject “we’ve been doing a lot of sinning lately”. Well, I finally had a chance to read it this evening, and I’m glad I got to it late, because it was the comments I found the most illuminating. Paul Houle’s comment gave me the first glimpse of a reality where unit testing just might not apply (in the same way) to web applications.
In some applications, objects are self-contained, activities are sequential, and algorithms are tricky. Automated unit testing is cheap and beneficial. You may have already spent the cost that it takes to break the system into testable units, and know how to make a good architecture with those constraints.
In other cases (say a browser-based app) you’re writing simple programs that work by causing side effects in a large, complex and unreliable external system. In that case the real issue is “does the external system behave the way I want it to” and the nature of your testing is entirely different.
Here’s another tidbit I stumbled across on Ward Cunningham’s patterns wiki:
Once again, web applications AND databases used in a single sentence to describe things that are hard to test, and when they’re restructured to improve testability, they end up needlessly complicated. I swear, I went looking for resources to help me improve web app unit testing, and I ended up with the sense that what I’m looking for might not even be possible.
Update, July 27, 2009: There was another blog post I stumbled upon when I was writing this, and it’s stuck in my mind since then, so I’m going to include an excerpt here. It’s by this guy named Hamlet D’Arcy, called Big Flat Test Cases Suck (and what to do about it):
Beginners always created a test case per class; there was a one to one mapping between production classes and test classes, and I’d guess most people are still doing this.
The enlightened testers were creating one test case per fixture, in which test methods were all in the same test class if they shared the same setUp() method…
The really cool kids, however, were doing test case per feature. This entails grouping test methods together based on what feature or user story they implement.
But don’t be fooled! All three of these test organization methods are essentially the same…Test cases grow. They grow really long. And they all grow in one direction: down the screen.
At least I can skip ahead to the “really cool kids” part!
have you looked into what Rails does here? I haven’t ever had the pleasure of working on a Rails app (yet), but from the odd glance, it appears they might have some very good ideas:
http://guides.rubyonrails.org/testing.html
I looked into that a little. all in all, I think the key approach is to:
1. consider the database and its contents part of your app, and part of what’s tested. so flush it out and replace with nothing but known-good test data first.
2. do end-to-end tests that traverse the HTTP controller layers, instead of short-circuiting beneath them.
Once you take those into account, though, you move further from unit testing into system test territory. personally I think that’s unavoidable, and my use of unit tests recently has been either for self-contained library classes, or via a mess of nasty mock objects. (hate them)
The next step btw is system tests of _entire server networks_. deploy the app and its components to a collection of EC2 nodes, and test the lot as a single tested entity. fun!
SeleniumRC and Selenium IDE. I’ll be happy to give more details soon. ;-)
Yeah, it’s hard. And anyone who says otherwise is lying. The #1 thing you can do is build good tests for the business logic. Either abstract the business object code from the database layer (via mock objects), or else grit your teeth and build a little custom test database with known, simple data. Then test the hell out of your business logic code.
Selenium looks promising for testing the HTML/CSS/Javascript presentation code. Ideally you’d be testing that in isolation from the business logic and database.
Justin M, nope, I really haven’t spent much time with Rails either. Thanks for the link, I’ll check it out.
You and Nelson seem to confirm (along with the excerpts at the bottom of the post) that unit testing web apps is much easier said than done. Which makes me feel better about having had such a hard time trying to figure out how to do it.
I just haven’t really seen anyone of significant stature say, “unit testing web applications is practically impossible, so don’t do it. instead do X”. It’s the “instead do X” that I’m really seeking. People have bandied about terms like “functional testing” and “end-to-end testing” but without any code samples that makes those things real to me.
Heh, “Da Bull”, I almost name-dropped Selenium at the end of the post, but figured, I don’t know enough about it to dismiss it yet. It just kind of gives me the heebie jeebies, at least from a maintainability perspective. I figure for things that aren’t changing much, it could be cool. But that just hasn’t been the case here.
unit tests aren’t intended for UIs. if your web app has an api layer (which might correspond to the controller in a MVC model), then you can write unit tests, since the api will (theoretically) always respond accordingly.
at work, our application is tiered into two apps: the business logic backend, whose only output is XML (with HTTP headers), and our UI front-end, which is built on top of that API. we have unit tests which do functional testing against the API (and that’s mostly what you’ll read about), and we have a human team that goes in and does the UI testing – any time a change is made within an area of the UI, our testers go in and test against a growing list of use cases (and past regressions). this isn’t sexy, and it feels like there should be a ‘better’ solution, but with the complexities and subtleties of UeX, it’s actually more work to try to automate.
there are tools that do gui testing (you basically record a set of “clicks” which replay through the tool and compares the screens to ensure the same outcome), but we ended up getting more false positives.
We’ve built a testing framework that does inputs, clicks, and describes based on <label>s. Works pretty well for us. It runs on Selenium in our nightlies, but we also have an in-browser console that does everything in Javascript. (Rebinding to a page after clicking on a link is hard.)
I’m totally with Paul Houle and Nelson on this one. If you’re just testing your classes to make sure that no one has a typo (PHPUnit), you’re just testing your web-app layer. What the business really needs is for us programmers to start testing the business logic layer.
I would posit that most web-based apps don’t even have the business logic that they’re supposed to reflect written out anywhere to test against.
That is step #1: document what a working business-logic based system looks like.
Then I think what JWatt is getting at is what makes #2 hard: most systems have unpredictable data inputs, which makes static-data based unit tests not useful for testing the functionality of our business logic-layer.
So where to go from here? I think there’s a layer needed beyond PHPUnit that will simulate your ecosystem: replace all external agents with simulations that return known data… we need to abstract testing out beyond our own models.
But I would argue that we shouldn’t give up the goal of TDD because the current TDD best-practices don’t give us full coverage, or b/c they’re hard
There are many different types of testing, and there are many different types of development. You’re asking a few different questions here, but they seem to be blurring into each other. I’m going to try to separate them:
1. What is “test-driven development” (TDD)?
– When you want to write a new feature, you first write an automated test that fails until the new feature is implemented. In TDD, you always write tests before you write code.
2. What kinds of testing are there?
– Unit test: a test that exercises the smallest functional unit of your code. In most languages, this means either a single method of a single class, or a single freestanding function (aka “a unit”). If your code is procedural, it will need to be refactored. If your unit relies on external dependencies, such as a database, or a filesystem, or *any other unit of code*, it will need to be mocked, faked, or stubbed. If your code wasn’t written in the first place with the intent of being tested, it’ll probably need a little refactoring to allow for dependency injection. The important thing is that your unit test should execute as little code as possible outside of the unit being tested.
– Integration test: any test that exercises two or more components of your system running together.
– Functional test: a type of integration test that exercises a specific piece of functionality within your system. All other things should be mocked, faked, or stubbed as necessary.
– End-to-end test: a test that exercises important functionality within a complete system. These are usually performed on staging environments before a build would be pushed to prod. This is technically a type of functional test where the “function” is “everything the app does”.
3. How can a person apply TDD to web development?
– Say you want to create a widget that displays a whodiddly whenever it’s clicked. Write a functional (e.g. in Selenium) test (or tests) to test what that the widget exists and behaves correctly. The test should fail until the widget displays the whoddidly properly.
Most likely, by this point you’re already asking yourself “But I haven’t written it yet. I’m sure I’ll go through several iterations before I know what ‘properly’ even means.” Three ways to approach this:
I. Write the test against what you currently think will end up being the implementation. As you iterate on your design, update the test.
II. Design harder before writing any code so that you can write tests you won’t have to change. Few people are this rigorous.
III. Don’t use TDD. By far, a common approach. You don’t have to write the tests first to have a well-tested application.
On a smaller scale and entirely independent of whether you choose to employ TDD on a functional level, you may also employ TDD to write unit tests for your functions and classes.
4. What good are unit tests if they don’t test the db and the filesystem?
– This really depends on who you are, who you’re working with, how complicated your application is, what your development cycle looks like, your level of code reuse, and a whole host of other things. For some folks, unit testing makes code more understandable, or it makes it easier to see what example implementations might look like. Some teams need unit testing in order to make explicit details that otherwise might have been left out of the documentation. When we edit each others’ code (and often our own, after a period has passed), it helps to have a safety net so we don’t introduce unintended behavior.
Of course, all of these things can also be said for functional testing as well. For me, the benefit of unit testing is confident code reuse. If the Foo object has its methods well tested, I’m much more likely to be able to use it in both the Bar and Baz applications. Functional testing provides a great safety net to make sure your application as a whole works well, but it doesn’t tell you anything about the tiny pieces of code within.
In general, it’s a good idea to have lots of small tests, some medium tests, and a few large tests of application sanity.
Roy, I should add that the other outcome from our first foray into unit testing (besides one-command install) was much improved, object-oriented controllers. Of course it took some significant effort for that refactoring to work its way through the codebase, which is probably what sapped the energy from our nascent unit testing effort. Thanks for your comment, it’s cool to see how other groups structure themselves around this task. Btw, are you tabulas.com-Roy?
Another vote for Selenium from Braden. Good to know. Thanks.
JS, maybe I just hate documentation, but I’m getting to the point where I want to have the business logic written out in code, and consolidated in one place. It seems like this is where unit tests would really shine. There’s no one true spot for that when it comes to most implementations of MVC. As it happens, sometimes our business logic is in the model classes, sometimes it’s in the controllers.
Danny, wow. It’s not everyday I get a 700+ word comment. Thank you. In any case, you’re right, I do tend to conflate test-driven development and unit testing. Even just mentioning it in the title is putting the cart before the horse I suppose, as one would have to know how to write good tests before employing test-driven development as a process. On the other hand, employing test-driven development provides explicit motivation for writing tests, which might not happen in isolation. Anyway, test-driven development, as a concept, is not really what I’m after. I get test-driven development.
What I really want to explore is “how to automate web application testing” where that web app has a fairly vanilla MVC architecture with a database backend—what most people would call a content management system.
I need a character limit to keep me concise. :)
In that case, chalk another +1 up to Selenium for end-to-end tests and phpunit for the unit tests, fired automatically by pre- or post-commit hooks in your VCS.
sorry for the late response – (three weeks late?!) – but yes, this is tabulas roy :) (although i’m probably more of a “mindtouch” roy now!)
cool cool, i was just curious. sometimes it’s hard to know for sure without a URL pointing back to some sort of online identity… :)