Getting started with Autotest - Continuous Testing

Last update: Nov 11, 2008

1. Why Autotest?

1.1. Continuous Testing

The cool thing about Autotest is that you have instant feedback on your code (tests run within a second). Even better, the testing happens on its own so you no longer have to switch back and forth from the coding context to the testing context anymore (both wise cognitively and from a UI perspective). This effortless and immediate feedback on your code as well as the automated and unattended test runs are quite similar to the characteristics of Continuous Integration at the team level. However, continuous integration concentrates on improving integration at a team level while Autotest concentrates on facilitating the development for a single developer (or programming-pair) before the code gets integrated – hence the term Continuous Testing.

As this is highly visual, have a look at Nuby on Rails’ autotest screencast.

1.2. Quicker Test Runs

Autotest can also provide quicker test runs than standard convention since it intelligently monitors the changes and only runs the tests that are impacted by these changes. In practice this is relevant only for classic Rails applications because:

• Rails conventions provide good heuristics for Autotest to decide which tests to run when a file changes. If your application does not stick to the classic Rails layout and naming conventions the magic does not work so well anymore. In this case, it is probably better to have Autotest run your whole test suite for all changes.

• There is value in running a subset of the whole test suite only because running classic Rails unit tests can be slow. This is mostly since Rails approach to unit testing is quite unconventional and involves database access. This approach goes against general agile wisdom that you should make sure your unit tests run quickly, and do not have any dependency for external systems. Also note that there are well-documented ways to have your Rails unit tests not depend on the database and have them run blazing fast!

1.3. Make Up for a Lack of a Proper Ruby IDE

Autotest can come in handy if your favorite IDE has limited Ruby support, or if you prefer a more ligthweight development environment (text editor + terminal + Autotest): it gives you an easy and automated way to run your tests.

2. Install Autotest

2.1. Make Sure You Already have RubyGem Installed

The easiest way to install Autotest is to use the ZenTest gem. If you have no idea of what a gem is, or you have not installed the RubyGem packaging system yet, please have a look at the RubyGem official website. If you are serious about Ruby development, it will be hard not to use RubyGem.

2.2. OS X and Linux

On OS X or Ubuntu launch from a terminal:

$ sudo gem install ZenTest

3. Run Autotest

3.1. Ruby on Rails project

Consistent with Rails principles, Autotest does not require any configuration to run. Provided you follow classic Rails conventions, Autotest will figure things out on its own. Simply launch Autotest from the base directory of your Ruby on Rails project.

$ cd <base directory of your Ruby on Rails project>
$ autotest

Autotest will then run all your tests (the first time), and wait for you to modify some code:

$ autotest
/usr/bin/ruby1.8 -I.:lib:test -rtest/unit -e "%w[test/functional/tasks_controller_test.rb test/unit/quarter_test.rb test/unit/task_test.rb].each { |f| require f }" | unit_diff -u
Loaded suite -e
Started
.......................
Finished in 0.672928 seconds.

================================================================================
**23 tests, 60 assertions, 0 failures, 0 errors**

Go ahead and modify some code in your project so that a test fails. Save the modified file to disk and Autotest will automatically rerun some of the tests:

/usr/bin/ruby1.8 -I.:lib:test -rtest/unit -e "%w[test/functional/tasks_controller_test.rb test/unit/task_test.rb].each { |f| require f }" | unit_diff -u
Loaded suite -e
Started
...F........
Finished in 0.42272 seconds.

1) Failure:
test_should_be_found(TaskTest) [**./test/unit/task_test.rb:22**]:
--- /tmp/diff6647.0     2006-11-15 20:46:43.000000000 -0800
+++ /tmp/diff6647.1     2006-11-15 20:46:43.000000000 -0800
@@ -1 +1 @@
**
-Expected result
+Actual result**

================================================================================
**4 tests, 9 assertions, 1 failures, 0 errors**</pre>

Note that autotest ran only a subset of your test suite this time (4 tests out of 23 in my case). Also note that Autotest is especially good in providing brief and relevant feedback on the test failures.

Autotest focuses on running previous failures until you have fixed them. So test failures are run until they have all passed. Then the full test suite is run to ensure that nothing else was inadvertently broken.

3.2. Ruby Project

In theory you would run Autotest the same way as you would for any Ruby project – even if it is not based on Rails:

$ cd <base directory of your Ruby project>
$ autotest

In practice, Autotest might have problems finding your tests or figuring out which tests to run when you change some code. If this is the case, take a look at the troubleshooting test detection section.

3.3. Forcing a Full Test Run and Stopping Autotest

If you want to force Autotest to run the entire test suite hit Ctrl - C once in the terminal running Autotest. Hitting Ctrl - C twice will stop Autotest.

4. Configure Plugins

Autotest also provides some cool plugins that enable you to get feedback the way you want.

4.1. Create a file

You configure plugins by creating a .autotest file in your project base directory. You can also provide a default configuration for all your projects by creating a .autotest file in your home directory (when present, project configuration files override user default configuration file).

You enable a plugin by adding a line requiring the plugin in the .autotest file. For instance, to enable the Growl plugin, you would add the following line:

require 'autotest/growl'

Below you will find a description of the most popular plugins and how to enable them.

4.2. Red / Green Plugin

The Red / Green plugin provides color to Autotest messages in the terminal window. As expected, output is green if all the tests pass, red if some test fails. Having red / green visual output makes it easier for one to scan the output and quickly determine whether everything is OK or something went wrong.

Visually, the Red /Green plugin turns

================================================================================
200 tests, 520 assertions, 0 failures, 0 errors

into

================================================================================
200 tests, 520 assertions, 0 failures, 0 errors

or

================================================================================
5 tests, 20 assertions, 1 failures, 0 errors

To enable the Red / Green plugin add the following line to your .autotest file:

require 'autotest/redgreen'

4.3. Desktop Notification Plugins

You might not even have to look at the Autotest terminal output to figure out the result of a test run. Several plugins provide desktop notification messages capabilities. In this way you can run Autotest in the background and see popup messages when something fails.

4.3.1. Growl Plugin (OS X)

Growl is a popular desktop notification system for OSX. If you are developing on a Mac, enable the Growl plugin by adding

require 'autotest/growl'

to your .autotest file. Note that for this plugin to work, you not only need to install Growl but also its command line interface: growlnotify.

4.3.2. Snarl Plugin (Windows)

Snarl is a notification system for Windows largely inspired by Growl. Autotest will use it if you enable the Snarl plugin in your .autotest file:

require 'autotest/snarl'

4.3.3. KDE Notify Plugin (Linux)

If you are running Linux and use KDE as your desktop environment, you will get desktop notification by adding to your .autotest file:

require 'autotest/kdenotify'

4.3.4. Gnome Notify plugin (Linux)

If you are running Linux and use Gnome as your desktop environment, unfortunately there is no official plugin for desktop notification. You can still get desktop notifications by adding the following code snipet in your .autotest file:

module Autotest::GnomeNotify

  # Time notification will be displayed before disappearing automatically
  EXPIRATION_IN_SECONDS = 2
  ERROR_STOCK_ICON = "gtk-dialog-error"
  SUCCESS_STOCK_ICON = "gtk-dialog-info"

  # Convenience method to send an error notification message
  #
  # [stock_icon]   Stock icon name of icon to display
  # [title]        Notification message title
  # [message]      Core message for the notification
  def self.notify stock_icon, title, message
    options = "-t #{EXPIRATION_IN_SECONDS * 1000} -i #{stock_icon}"
    system "notify-send #{options} '#{title}' '#{message}'"
  end

  Autotest.add_hook :red do |at|
    notify ERROR_STOCK_ICON, "Tests failed", "#{at.files_to_test.size} tests failed"
  end

  Autotest.add_hook :green do |at|
    notify SUCCESS_STOCK_ICON, "All tests passed, good job!", ""
  end

end

For this to work you need to have libnotify installed on your system and a program named notify-send in your PATH. For most Linux distributions this simply means that you should install the libnotify-bin package. If you are running Ubuntu, run

sudo apt-get install libnotify-bin

4.4. Pretty Plugin

If you are running Autotest on a Mac you can enable the pretty plugin to visualize your Autotest status history as a sequence of red and green squares:

<p class="figure">
  <img alt="Autotest Pretty Plugin Screenshot" src="../../images/Autotest Pretty Plugin.png" />
</p>

Of course a green square indicates passing tests while a red square signals some test failures. If you want to get a feel of what a session is like with this plugin enabled, ZenSpider has a nice video demonstrating the pretty plugin in action.

To enable this plugin you will need to install RubyCocoa and add the folowing line to your .autotest file:

require 'autotest/pretty'

4.5. HTML Report Plugin

The HTML report plugin publishes most recent Autotest statuses as a web page under ~/Sites/autotest.html. You can then point a browser to this page and see something like:

Of course the content of the web page is updated while you work. Note that you need to have an an existing ~/Sites directory before you enable the plugin with the following line:

require 'autotest/html_report'

4.6. Menu Plugin

Remember that, by default, hitting Ctrl - C once will force Autotest to run the entire test suite, and hitting Ctrl - C twice will stop Autotest? The menu plugin changes this behavior: each time you hit Ctrl - C it will explicitly ask you whether you want to quit, restart or just keep going.

c: continue q: quit r: restart menu>

Enable the menu plugin by adding the following line to your .autotest file.

require 'autotest/menu'

4.7. Timestamp Plugin

While Autotest waits for you to save a file, the timestamp plugin prints a message with the current time. Messages look like:

# waiting... Sat Feb 03 15:56:23 EST 2007

To enable the timestamp plugin add the following to your .autotest file:

require 'autotest/timestamp'

4.8. Getting More Information

Your Autotest install comes with a sample .autotest file listing all available plugins. It is named example_dot_autotest.rb. You will find it in the gems install directory. Most likely this directory will look like:

• /Library/Ruby/Gems/1.8/gems/ZenTest-3.4.3/ on OS X.
• /usr/lib/ruby/gems/1.8/gems/ZenTest-3.4.3/ on other Unix platforms

5. Troubleshooting Autotest Test Detection

Whether Autotest does not work out of the box for you or its magics eludes you, it is always good to get some understanding of the heuristics that Autotest uses to figure which test(s) to run.

5.1. Heuristics for Rails

Autotest automatically discovers Ruby on Rails projects by checking for a config/environment.rb file. If there is one, Autotest will base its logic on standard Rails file mappings and conventions.

If for some reason you want to force Ruby on Rails mode you can always launch Autotest it with the -rails option:

$ autotest -rails

A simplified version of Autotest heuristics in this mode would be:

• When changing a test file, only this file is run (e.g. test/unit/foo_test.rb → test/unit/foo_test.rb).

• When changing a model file, only associated unit test file is run (e.g. app/models/foo.rb → test/unit/foo_test.rb).

• When changing a controller file, associated functional test file is run (e.g. app/controllers/foo_controller.rb → test/functional/foo_controller_test.rb).

• When changing a fixture file, associated unit test and functional test are run (e.g. app/fixtures/foos.yml → test/unit/foo_test.rb + test/functional/foo_controller_test.rb).

• When changing a helper file, associated functional test file is run (e.g. app/helpers/foo_helper.rb → test/functional/foo_controller_test.rb).

• When changing application_helper.rb file all functional test files are run (e.g. application_helper.rb → test/functional/*_test.rb).

• When changing a file under the config directory, all tests are run.

You’ve got the idea. Actual heuristics are a little more complex and also handle the concept of view and controller tests. For a more thourough understanding have look at the rails_autotest.rb file in ZenTest gem install directory.

In case these heuristics do not play well with your own conventions, do not give up yet: you can always configure Autotest to run the whole test suite for all changes.

5.2. Heuristics for Non Rails Projects

For non Rails project, Autotest uses a simple naming scheme to map implementation files to test files:

• Test files must be stored in the test directory

• Implementation files must be stored in the lib directory

• Test files names must start with test_

• Test files corresponding to a specific implementation file must be named test_*name_of_implementation_file.rb

• Test class names must start with Test

• The test class name corresponding to a specific implementation class must be named TestImplementationClassName.rb

If you can live with these conventions, Autotest will work out-of-the-box for you. If these conventions are not your cup of tea and you have your own, the next paragraph explains how to configure Autotest so that it runs the whole test suite each time you save a file.

5.3. Running the Whole Test Suite for All Changes

If for some reason Autotest heuristics do not work for you, you can customize them in your .autotest file with a little bit of work.

For instance, if your entire test suite runs quickly (as it should), you can easily configure Autotest to run the whole test suite for any change by adding the following code to your .autotest file:

# Override autotest default magic deciding which test to run when
# a file is changed : enable more flexible naming conventions
# trading some of the efficiency: we rerun all the tests each time.
#
class Autotest

  def tests_for_file(filename)
    return Dir["test/**/*.rb"]
  end 

end

6. Conclusion

Autotest provides an easy and effortless way to run your tests: just save a file. This is a great way to get quick feedback on your code and avoid any context switch. Autotest automated test runs are also extremelly valuable if your favorite IDE has poor Ruby support, or if you prefer a more ligthweight development environment (text editor + terminal + Autotest).

Autotest also tries hard to be smart on deciding which tests to run:

• It only runs the tests affected by your latest code changes.

• When some tests fail, Autotest focuses on running previous failures until you have fixed them. Once they pass, the full test suite is run to ensure that nothing else was inadvertently broken.

On deciding which tests to run, Autotest magic works out of the box if your application follows classic Ruby on Rails conventions. If this is not your cup of tea, it is extremely easy to customize Autotest to fit your conventions.

Via its plugins Autotest also offers a lot of interesting feedback options, from terminal output to html publishing to desktop notifications. The pretty plugin offers a highly visual representation of your test runs history, which could be useful when teaching Test Driven Development (green, red, green, red, …).

On the flip side, It is important to note that Autotest does not fit all developement styles: Some developers like better control on which tests they are running. While working on a piece of code, they will typically focuss on a few tests (which they know they could have broken), and then run the whole test suite just before committing. Autotest emulates this as well as it can with his focus on running previous failures; but ultimately a human will always have a better intuition, especially if your project does not follow classic Rails conventions.

In all cases, it is worth spending some time playing with Autotest and experiment with its innovative, lightweight and effortless approach to test runs, what I have been calling continuous testing.