Getting Started with Autotest – Continuous Testing

First Published: Jan 2007
Last update: Dec 2009

Why manually run your tests, when the computer can do it for you! Autotest is a great tool to speed up test-driven development with Ruby or Ruby on Rails.

Autotest makes your coding session even more productive as it automatically runs a subset of your test suite each time you change a file. Autotest is smart – it figures out which subset to run based on the files you've changed. Think of it as Continuous Testing.

1. Why Autotest?

1.1. Continuous Testing

The cool thing about Autotest is that you have instant feedback on your code (tests run within a second). Even better, the testing happens on its own so you no longer have to switch back and forth from the coding context to the testing context anymore (both wise cognitively and from a UI perspective). This effortless and immediate feedback on your code as well as the automated and unattended test runs are quite similar to the characteristics of Martin Fowler’s legendary introduction to Continuous Integration at the team level. However, continuous integration concentrates on improving integration at a team level while Autotest concentrates on facilitating the development for a single developer (or programming-pair) before the code gets integrated – hence the term Continuous Testing.

As this is highly visual, have a look at Nuby on Rails’ autotest screencast.

1.2. Quicker Test Runs

Autotest can also provide quicker test runs than standard convention since it intelligently monitors the changes and only runs the tests that are impacted by these changes. In practice this is relevant only for classic Rails applications because:

1.3. Make Up for a Lack of a Proper Ruby IDE

Autotest can come in handy if your favorite IDE has limited Ruby support, or if you prefer a more lightweight development environment (text editor + terminal + Autotest): it gives you an easy and automated way to run your tests.

2. Install Autotest

2.1. Make Sure You Already have RubyGem Installed

The easiest way to install Autotest is to use the ZenTest gem. If you have no idea of what a “Ruby Gem” is, or you have not installed the RubyGem packaging system yet, please have a look at the RubyGem official website.

Trust me, if you are serious about Ruby development, it will be hard not to use RubyGem.

2.2. OS X and Linux

On OS X or any other UNIX systems you typically install ZenTest with the following command:

sudo gem install ZenTest

If you are going to do any Ruby on Rails development, you also need to install the autotest-rails plugin:

sudo gem install autotest-rails

3. Run Autotest

3.1. Ruby on Rails project

Consistent with Rails principles, Autotest does not require any configuration to run. Provided you follow classic Rails conventions, Autotest will figure things out on its own. Simply launch Autotest from the base directory of your Ruby on Rails project.

$ cd <base directory of your Ruby on Rails project>
$ autotest

Autotest will then run all your tests (the first time), and wait for you to modify some code:

$ autotest
/usr/bin/ruby1.8 -I.:lib:test -rtest/unit -e "%w[test/functional/tasks_controller_test.rb test/unit/quarter_test.rb test/unit/task_test.rb].each { |f| require f }" | unit_diff -u
Loaded suite -e
Started
.......................
Finished in 0.672928 seconds.

================================================================================
**23 tests, 60 assertions, 0 failures, 0 errors**

Go ahead and modify some code in your project so that a test fails. Save the modified file to disk and Autotest will automatically rerun some of the tests:

/usr/bin/ruby1.8 -I.:lib:test -rtest/unit -e "%w[test/functional/tasks_controller_test.rb test/unit/task_test.rb].each { |f| require f }" | unit_diff -u
Loaded suite -e
Started
...F........
Finished in 0.42272 seconds.

1) Failure:
test_should_be_found(TaskTest) [**./test/unit/task_test.rb:22**]:
--- /tmp/diff6647.0     2006-11-15 20:46:43.000000000 -0800
+++ /tmp/diff6647.1     2006-11-15 20:46:43.000000000 -0800
@@ -1 +1 @@
**
-Expected result
+Actual result**

================================================================================
**4 tests, 9 assertions, 1 failures, 0 errors**</pre>

Note that Autotest ran only a subset of your test suite this time (4 tests out of 23 in my case). Also note that Autotest is especially good in providing brief and relevant feedback on the test failures.

Autotest focuses on running previous failures until you have fixed them. So test failures are run until they have all passed. Then the full test suite is run to ensure that nothing else was inadvertently broken.

3.2. RSpec Integration

Autotest assumes by default that you are writing your tests using Test::Unit, or a compatible test framework (like Shoulda or minitest).

If you prefer to write your tests the RSpec way, you will need to give Autotest a little hint to trigger the integration with RSpec1: Set the RSPEC environment variable to true before invoking the autotest command:

RSPEC=true autotest

Alternatively RSpec ships with a autospec command that you can use as a drop-in replacement of autotest to automate the running of your specs:

autospec

3.3. Ruby Project

In theory you would run Autotest the same way as you would for any Ruby project – even if it is not based on Rails:

cd <base directory of your Ruby project>
autotest

In practice, Autotest might have problems finding your tests or figuring out which tests to run when you change some code. If this is the case, take a look at the section on “Troubleshooting Autotest Test Detection

3.4. Forcing a Full Test Run and Stopping Autotest

If you want to force Autotest to run the entire test suite hit Ctrl-C once in the terminal running Autotest. Hitting Ctrl-C twice will stop Autotest.

4. Enable Plugins and Customize Autotest

Autotest also provides some cool plugins that enable you to get feedback the way you want.

4.1. Create a .autotest file

You customize Autotest and configure plugins by creating a .autotest file in your project base directory. You can also provide a default configuration for all your projects by creating a .autotest file in your home directory. As you would expect, when present, project configuration files override user default configuration file.

Once you have a .autotest file, enabling a new autotest plugin on your system for the first time typically involves 2 steps:

sudo gem install autotest-growl
require "autotest/growl"

Keep in mind that you have to restart autotest after you edit the .autotest file before the changes take effect though2.

Below you will find a description of the most popular plugins and how to enable them.

4.2. FSEvent Plugin

Part of Autotest’s smarts is that it automatically detects your changes as soon as you save a file. However, if you go with Autotest’s vanilla install, this magic comes with a high price tag. Autotest will constantly poll your filesystem to detect changes: a sure way to peg your CPU, drain your battery in no time, fry your hard-drive, increase your carbon footprint… well you get the picture! In short, it is bad Karma, and the only reason Autotest does it this way by default is portability: This implementation does not make any assumption on the capabilities of your operating system.

Modern operating systems provide filesystem event notification APIs which provide a much more efficient way to detect changes on the filesystem3. In a nutshell, an application can register interest for specific events and specific areas of the filesystem. The O.S. will then alert the application via asynchronous callbacks when (and only when) changes happen on the filesystem.

This is why you should install the autotest-fsevent plugin if you are developing on a Mac: once you install and enable autotest-fsevent, Autotest will stop constantly polling your filesystem and will use OS X filesystem event notifications to detect changes. Do it for your Mac, it will give it back to you!

First install the plugin:

sudo gem install autotest-fsevent

Then enable it in your .autotest file:

require 'autotest/fsevent'

And voila, Autotest is now gentle with your computer! There are actually ways to make it even more gentle… this is the topic of the next section.

4.3. Ignore Files With Exceptions

By default, Autotest will monitor every file in your ruby/rails project for changes4. By telling it to ignore certain files, you can significally reduce its impact on the CPU, disk and battery life.

You can add explicit exceptions (files to be ignored) in your .autotest file with the add_exception command. For instance to ignore the content of a various source control meta data, some OS X meta files and the vendor5 directory for your Ruby project, add the following code to your .autotest file:

Autotest.add_hook :initialize do |autotest|
  %w{.git .svn .hg .DS_Store ._* vendor tmp log doc}.each do |exception|
    autotest.add_exception(exception)
  end
end

4.4. Restart Plugin

When you are intensively tweaking your .autotest file – as you probably are while reading this article – manually restarting autotest after each change to .autotest can be a drag. Enter the restart plugin: when you enable it, Autotest will restart itself automatically as soon as a change in the .autotest file is detected.

To enable the plugin, all you need to do is add the following snippet to your .autotest file:

require "autotest/restart"

… and restart autotest manually one last time (ironically).

4.5. Red / Green Plugin

The “Red/Green” plugin provides color to Autotest messages in the terminal window. As expected, output is green if all the tests pass, red if some test fails. Having red / green visual output makes it easier for one to scan the output and quickly determine whether everything is OK or something went wrong.

Visually, the Red /Green plugin turns

================================================================================
200 tests, 520 assertions, 0 failures, 0 errors

into

================================================================================
200 tests, 520 assertions, 0 failures, 0 errors

or

================================================================================
5 tests, 20 assertions, 1 failures, 0 errors

Install the plugin with:

sudo gem install redgreen

To enable the “Red/Green” plugin add the following line to your .autotest file:

require 'redgreen/autotest'

4.6. Desktop Notification Plugins

You might not even have to look at the Autotest terminal output to figure out the result of a test run. Several plugins provide desktop notification messages capabilities. In this way you can run Autotest in the background and see popup messages when something fails.

4.6.1. Growl Plugin (OS X)

Growl is a popular desktop notification system for OS X. If you are developing on a Mac, install the Growl plugin:

sudo gem install autotest-growl

Then enable it by adding

require 'autotest/growl'

to your .autotest file.

Note that for the Growl plugin to work, you need to have Growl itself already installed on your system. If you did not install Growl or are unsure, download the latest version from the Growl Website.

The Growl plugin offers a few customization options that you can set in your .autotest file. They are documented in the README.rdoc file included in the gem. Here are a few settings you can tweak:

Autotest::Growl::show_modified_files = true
Autotest::Growl::one_notification_per_run = true
Autotest::Growl::clear_terminal = false
Autotest::Growl::hide_label = true

4.6.2. Test Notifier Plugin (Linux and Windows)

If you are running Linux or Windows, you can still get desktop notifications with the test notifier plugin. This plugin provide desktop notifications for Linux, Windows and OS X using a broad range of native technologies (libnotify, knotify, Snarl and Growl just to name a few). Note that on OS X, the Growl plugin – which we just described in the previous section – provides a more lightweight install than the test notifier plugin though. So in this document I will only cover installation on Linux and Windows6.

Before installing the test notifier plugin you will need to install libraries for the underlying notification mecanism the plugin will trigger:

sudo apt-get install libnotify-bin
sudo apt-get install xosd-bin`
gem install ruby-snarl

Once you are done installing the underlying native notification system library you can install and enable the test notifier plugin the usual way. First:

sudo gem install test_notifier

Then add the following code snipet in your .autotest file:

require "test_notifier/autotest"

4.7. Timestamp Plugin

While Autotest waits for you to save a file, the timestamp plugin prints a message with the current time. Messages look like:

# waiting... Sat Feb 03 15:56:23 EST 2007

To enable the timestamp plugin add the following to your .autotest file:

require 'autotest/timestamp'

4.8. Getting More Information

Your Autotest install comes with a sample .autotest file listing all available plugins. It is named example_dot_autotest.rb. You will find it in the gems install directory. Most likely this directory will look like:

5. Troubleshooting Autotest Test Detection

Whether Autotest does not work out of the box for you or its magics eludes you, it is always good to get some understanding of the heuristics that Autotest uses to figure which test(s) to run.

5.1. Heuristics for Rails

Autotest automatically discovers Ruby on Rails projects by checking for a config/environment.rb file. If there is one, Autotest will base its logic on standard Rails file mappings and conventions.

If for some reason you want to force Ruby on Rails mode you can always launch Autotest it with the -rails option:

$ autotest -rails

A simplified version of Autotest heuristics in this mode would be:

You’ve got the idea. Actual heuristics are a little more complex and also handle the concept of view and controller tests. For a more thourough understanding have look at the rails_autotest.rb file in ZenTest gem install directory.

In case these heuristics do not play well with your own conventions, do not give up yet: you can always configure Autotest to run the whole test suite for all changes.

5.2. Heuristics for Non Rails Projects

For non Rails project, Autotest uses a simple naming scheme to map implementation files to test files:

If you can live with these conventions, Autotest will work out-of-the-box for you. If these conventions are not your cup of tea and you have your own, the next paragraph explains how to configure Autotest so that it runs the whole test suite each time you save a file.

5.3. Running the Whole Test Suite for All Changes

If for some reason Autotest heuristics do not work for you, you can customize them in your .autotest file with a little bit of work.

For instance, if your entire test suite runs quickly (as it should), you can easily override Autotest’s default logic and configure it to run the whole test suite for any change by adding the following code to your .autotest file:

# Override autotest default magic to rerun all tests every time a
# change is detected on the file system.
class Autotest

  def get_to_green
    begin
      rerun_all_tests
      wait_for_changes unless all_good
    end until all_good
  end

end

6. Conclusion

Autotest provides an easy and effortless way to run your tests: just save a file. This is a great way to get quick feedback on your code and avoid any context switch. Autotest automated test runs are also extremelly valuable if your favorite IDE has poor Ruby support, or if you prefer a more ligthweight development environment (text editor + terminal + Autotest).

Autotest also tries hard to be smart on deciding which tests to run:

On deciding which tests to run, Autotest magic works out of the box if your application follows classic Ruby on Rails conventions. If this is not your cup of tea, it is extremely easy to customize Autotest to fit your conventions.

Via its plugins Autotest also offers a lot of interesting feedback options, from terminal output to html publishing to desktop notifications.

On the flip side, It is important to note that Autotest does not fit all developement styles: Some developers like better control on which tests they are running. While working on a piece of code, they will typically focuss on a few tests (which they know they could have broken), and then run the whole test suite just before committing. Autotest emulates this as well as it can with his focus on running previous failures; but ultimately a human will always have a better intuition.

In all cases, it is worth spending some time playing with Autotest and experiment with its innovative, lightweight and effortless approach to test runs, what I have been calling continuous testing.


  1. Assuming you already have the rspec gem already locally installed .

  2. Unless you enable Autotest restart plugin that we will cover later.

  3. S.G.I had FAM, Linux has inotify, OS X has FSEvent, etc.

  4. via FSEvent or brute force polling.

  5. The vendor directory is a great candidate for Autotest’s exceptions: You are not supposed to edit its content manually and it typically include a very large number of files.

  6. Test notifier README file contains installation instruction for OS X, if you really need them.

Original web site design by: JFX diz*web.