Acceptance tests are exams for developers

I had a great evening of drinking and talking (thanks Jared & other guys). Jared is a kick-ass tester, and has a great handle on the whole agile thing. It’s really awesome to spend time talking to people who actually “get it”.
Anyway, Jared provided this great analogy about doing acceptance testing. Acceptance tests are exams for the developers. They get the exams up front, they get to see the questions and spend time preparing the answers. Of course, the answers are obvious, so developers will know when they are right or wrong.
Now, this is really cool, because who’d think of giving an exam to people without providing them the questions ? You’d have all these developers writing answers that will hopefully be to the questions that the exam will contain, and you’ll have a good idea, but you’ll never know the exact questions you need to answer. It appears that plenty of software projects do this, and it’s broken, and wrong.
So developers, make sure you know the questions in your exam, before you start, or otherwise, you’ll never know what answers you need to provide.


2 thoughts on “Acceptance tests are exams for developers

  1. Hi Jon,
    I don’t like this analogy – it seems to suggest that
    all the acceptance tests need to be written and provided up-front, before development starts, like exam questions are written before an exam.
    The problem I have here is that detailed requirements as captured in acceptance tests are usually not fully understood prior to development. Developers in an Agile project are continually discovering gaps, inconsistencies and mistakes in requirements. (The requirements elicitation process continues during development and beyond). I don’t want to be in the situation where dealing with these discoveries is hard because those providing the tests wrote them too long ago and are now too busy trying to get the next set ready in time.
    Baically I dislike this analogy because examiners
    generally are teaching examinees and know the right questions to ask, wheras in software development both groups are learning together.
    I think its more like a study group than an exam…

  2. Thanks for that feedback Perryn. Maybe I should have been clearer. It’s not an up-front thing before any development starts.
    Provided the Acceptance Tests are ready for when the developers start developing a story/usecase/whatever that’s sufficient time.
    I do think that’s really important however, if the tests aren’t ready before the story starts, how will the developer know if they are finished ?
    Acceptance tests should be testing the business outcomes, and if that cannot be articulated by the business, well, then they don’t understand why this functionality is being built, and maybe it shouldn’t be in that case.
    If the business doesn’t understand enough, and wants to explore, then that’s also OK. They just write the criteria for what they understand now, let the developers develop the software, and then review the results. The tests can change, but that’s probably a good sign that the scope is changing as well. (That’s not a bad thing). The business can write more acceptance tests and the developers can implement the (new or changed) story.
    Without pre-written acceptance tests, developers will be guessing what the outcome is required by trying to read and evaluate a story. Of course, the alternative is to have a business owner responsible for that story sitting with the developer 100% of the time, to guide them on each step.
    Personally, I think pre-written acceptance tests are vital to the successful outcome of a project. It’s additional communication between the customer and the developer, and guides the customer to be far more explicit in describing how they want the software to work. And it’s done in a manner where the success criteria is relatively clear.

Comments are closed.