Here is an introduction video to OJAutotest. I've been working on this the past couple of days in order to make my life easier (I'm about to go into super-testing mode for our application).

OJAutotest Demo from Derek Hammer on Vimeo.

The important part to know here is that this is very much a convention-based tool. Your tests must be in Test and you must run ojautotest from the top level. If you are interested in using this tool, it is available on GitHub.