Effortless Thread Dump for Ruby:
Dump the Stack Trace of All the Threads in your Ruby Interpreter with caller_for_all_threads (M.R.I. 1.8)

caller for all threads sign
First published: May 2008
Last update: Dec 2008

By default, the Standard Ruby Interpreter (M.R.I) only gives you access to the stack trace of the current thread (Kernel#caller). Under these conditions, troubleshooting an unresponsive Ruby process in production can end up being quite challenging.

Enter caller_for_all_threads: just send a signal to your Ruby Interpreter to dump a stack trace for all the threads, getting a comprehensive view of your application state.

1. Enabling caller_for_all_threads support

caller_for_all_threads functionality needs to be supported at the Ruby interpreter level and cannot be implemented as a native gem: To gather the stack trace for all threads you need to be able to switch between the different (green) thread contexts. Doing so requires a lot more “intimacy” with the Ruby interpreter (and its setjmp / longjmp handling) than can be provided by a native gem.

So you will need to use a Ruby Interpreter that has been patched to provide support for caller_for_all_threads. If this sounds scary to you, no need to worry. There is an extremely easy and straightforward way to get a Ruby interpreter with caller_for_all_threads support: simply install Ruby Enterprise Edition.

1.1. Installing Ruby Enterprise Edition

Download Ruby Enterprise Edition and follow the installation instructions documented on the download page:

cd /tmp
curl -LO http://rubyforge.org/frs/download.php/47937/ruby-enterprise-1.8.6-20081205.tar.gz
tar zxvf ruby-enterprise-1.8.6-20081205.tar.gz
cd ruby-enterprise-1.8.6-20081205
./installer

That’s it! You now have a caller_for_all_threads-enabled Ruby interpreter installed under /opt/ruby-enterprise-1.8.6-20081205 (unless you explicitly changed the installation directory).

1.2. Compiling Your Own Ruby Interpreter

In case you prefer to compile your own Ruby interpreter from scratch, this is possible too. Apply the [ patch][caller_for_all_threads_patch] to M.R.I 1.8.6 or 1.8.7.

If you need more guidance on how to patch and build a Ruby interpreter from source, please refer to the appendix which provides step by step instructions. As you will soon realize, it is actually quite easy!

1.3. Setting Your Environment

Once you have a caller_for_all_threads-enabled Ruby interpreter, you need to make sure that you are going to use this particular Ruby install for the rest of this article (and not another one installed on your system). The simplest way to achieve this is to set your environment accordingly:

  export PATH=/opt/ruby-enterprise-1.8.6-20081205:${PATH}
  export LD_LIBRARY_PATH= /opt/ruby-enterprise-1.8.6-20081205/lib:${LD_LIBRARY_PATH}

You can then check that your environment is properly configured by running:

which ruby

which should return something like
/opt/ruby-enterprise-1.8.6-20081205/bin/ruby

2. Dumping the Stack Traces

Now that we have a caller_for_all_threads-enabled Ruby Interpreter, getting a thread dump is dead simple. Even a caveman can do it!

2.1. It’s on Kernel, Stupid!

caller_for_all_threads is just a method on Kernel. So you could potentially call it from anywhere in your code.

We can convince ourselves with a little irb session:

$ irb
irb(main):001:0> caller_for_all_threads
=> {#<Thread:0x22c6a8 run>=>["(irb):1:in `irb_binding'",
                             "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52:in `irb_binding'",
                             "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52"]}

irb(main):002:0> Thread.new { sleep 30; putc '.' }
=> #<Thread:0x2e4564 sleep>

irb(main):003:0> caller_for_all_threads
=> {#<Thread:0x2e4564 sleep>=>["(irb):2:in `irb_binding'",
                               "(irb):2:in `initialize'",
                               "(irb):2:in `new'",
                               "(irb):2:in `irb_binding'",
                               "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52:in `irb_binding'",
                               "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52"],
    #<Thread:0x22c6a8 run>=>["(irb):3:in `irb_binding'",
                             "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52:in `irb_binding'",
                             "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52"]}
irb(main):004:0>

Of course what you really want, is to write no code at all, just send a signal to the Ruby process and get a freshly baked stack trace.

2.2. Installing a Thread Dump Signal Handler

The quickest way to get a nice thread dump is to install a signal handler. You could do it manually with a handler like:

#
# Install a signal handler to dump backtraces for all threads
#
# Trigger it with: kill -QUIT <pid>
#
trap "QUIT" do
    STDERR.puts "\n=============== Thread Dump ==============="
    caller_for_all_threads.each_pair do |thread, stack|
      thread_separator = "-" * 78

      thread_description = thread.inspect
      thread_description << " [main]" if thread == Thread.main
      thread_description << " [current]" if thread == Thread.current
      thread_description << " alive=#{thread.alive?}"
      thread_description << " priority=#{thread.priority}"

      full_description = "\n#{thread_separator}\n"
      full_description << thread_description
      full_description << "\n#{thread_separator}\n"
      full_description << "    #{stack.join("\n      \\_ ")}\n"

      # Single puts to avoid interleaved output
      STDERR.puts full_description
    end
  STDERR.puts "\n==========================================\n"
end

Nevertheless the XRay gem provides a more convenient way to install a similar thread dump signal handler. First install the gem.

sudo gem install xray

Then, from your application code, require the file in charge of installing the thread dump signal handler:

require "rubygems"
require "xray/thread_dump_signal_handler"

In a Ruby on Rails application, you would typically require the thread dump signal handler in environment.rb or production.rb.

The XRay signal handler provides a nicely formatted dump, avoid interleaved output and and falls back on dumping only the current thread on platform where caller_for_all_threads is not supported.

2.3. Triggering the Thread Dump

At this point you can trigger a full thread dump, at any time, just by sending the QUIT signal to your Ruby process:

kill -QUIT <pid of your ruby process>

Let’s put it all together. Create a simple Ruby script under /tmp/sample.rb:

#!/usr/bin/env ruby

require "rubygems"
require "xray/thread_dump_signal_handler"

a_thread =Thread.new do
  loop { sleep 1; puts "." }
end

a_thread.join

Launch it with the (patched) Ruby interpreter:

ruby /tmp/sample.rb

Then open another terminal window and discover the pid of your Ruby process. For instance do this with:

$ ps auxww | grep sample.rb | grep -v grep
 ph7       9956   0.0  0.1    76892   2836 s002  S+    6:19PM   0:00.04 ruby /tmp/sample.rb

It is time to send the QUIT signal to this process to get our thread dump:

kill -QUIT 9956

At this point a nice thread dump should be ready for you in the terminal you started sample.rb in:

=============== XRay - Thread Dump ===============

#<Thread:0x255c9c sleep> alive=true priority=0

    /tmp/sample.rb:7
      \_ /tmp/sample.rb:7:in `loop'
      \_ /tmp/sample.rb:7
      \_ /tmp/sample.rb:6:in `initialize'
      \_ /tmp/sample.rb:6:in `new'
      \_ /tmp/sample.rb:6

#<Thread:0x22c6a8 run> [main] [current] alive=true priority=0

    /tmp/foo/lib/ruby/gems/1.8/gems/XRay-1.0.3/lib/xray/thread_dump_signal_handler.rb:9
      \_ /tmp/sample.rb:10:in `call'
      \_ /tmp/sample.rb:10:in `join'
      \_ /tmp/sample.rb:10

=============== XRay - Done ===============

Of course you can get more than one thread dump, and keep sending QUIT signals to get multiple thread dumps if this is what you want!

kill -QUIT 9956
kill -QUIT 9956
kill -QUIT 9956

A more realistic thread dump from a Rails application would look like:

=============== XRay - Thread Dump =========

 #<Thread:0x137158 run> [main] [current] alive=true priority=0

     /Users/ph7/Projects/Web-Site/vendor/gems/xray-1.1/lib/xray/thread_dump_signal_handler.rb:9
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:91:in `call'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:91:in `select'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:91:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:23:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:82:in `start'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/webrick_server.rb:62:in `dispatch'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/commands/servers/webrick.rb:66
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activesupport/lib/active_support/dependencies.rb:510:in `require'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activesupport/lib/active_support/dependencies.rb:355:in `new_constants_in'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activesupport/lib/active_support/dependencies.rb:510:in `require'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/commands/server.rb:39
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
       \_ ./script/server:3

 #<Thread:0x28d3c34 sleep> alive=true priority=0

     /Users/ph7/Projects/Web-Site/app/controllers/projects_controller.rb:11:in `index'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/base.rb:1166:in `send'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/base.rb:1166:in `perform_action_without_filters'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/filters.rb:579:in `call_filters'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/filters.rb:572:in `perform_action_without_benchmark'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/benchmark.rb:293:in `measure'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/rescue.rb:201:in `perform_action_without_caching'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb:13:in `perform_action'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:33:in `cache'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activerecord/lib/active_record/query_cache.rb:8:in `cache'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb:12:in `perform_action'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/actionpack/lib/action_controller/base.rb:529:in `send'
       #...

 #<Thread:0x2763ca0 sleep> alive=true priority=0

     /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/webrick_server.rb:77:in `service'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:162:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:95:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:92:in `each'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:92:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:23:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:82:in `start'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/webrick_server.rb:62:in `dispatch'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/commands/servers/webrick.rb:66
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activesupport/lib/active_support/dependencies.rb:510:in `require'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activesupport/lib/active_support/dependencies.rb:355:in `new_constants_in'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activesupport/lib/active_support/dependencies.rb:510:in `require'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/commands/server.rb:39
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
       \_ ./script/server:3

 #<Thread:0x2771c74 sleep> alive=true priority=0

     /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/webrick_server.rb:77:in `service'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:162:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:95:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:92:in `each'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:92:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:23:in `start'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/1.8/webrick/server.rb:82:in `start'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/webrick_server.rb:62:in `dispatch'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/commands/servers/webrick.rb:66
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activesupport/lib/active_support/dependencies.rb:510:in `require'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activesupport/lib/active_support/dependencies.rb:355:in `new_constants_in'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/activesupport/lib/active_support/dependencies.rb:510:in `require'
       \_ /Users/ph7/Projects/Web-Site/vendor/rails/railties/lib/commands/server.rb:39
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
       \_ /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
       \_ ./script/server:3

 =============== XRay - Done ===============

3. Conclusion

Hopefully you will find caller_for_all_threads useful for troubleshooting your production problems. I would love to hear from you and get your feedback: get in touch!

4. Appendix: Compiling Your Own Ruby Interpreter

4.1. Patching M.R.I 1.8

Go in a directory that you typically use to build software or to hack some code (for instance ~/projects) and execute the following commands:

For M.R.I 1.8.6:

  curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p114.tar.gz

  tar zxvf ruby-1.8.6-p114.tar.gz

  cd ruby-1.8.6-p114

  curl -o caller_for_all_threads_patch_for_MRI_1.8.6.diff ↩
          http://github.com/ph7/xray/tree/master/patches_for_mri/↩
                 caller_for_all_threads_patch_for_MRI_1.8.6.diff?raw=true

  patch -p0 < caller_for_all_threads_patch_for_MRI_1.8.6.diff

For M.R.I 1.8.7:

  curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz

  tar zxvf ruby-1.8.7-p72.tar.gz

  cd ruby-1.8.7-p72

  curl -o caller_for_all_threads_patch_for_MRI_1.8.7.diff↩
          http://github.com/ph7/xray/tree/master/patches_for_mri/↩
                 caller_for_all_threads_patch_for_MRI_1.8.7.diff?raw=true

  patch -p0 < caller_for_all_threads_patch_for_MRI_1.8.7.diff

That’s it! Now is time to compile the Ruby interpreter and install it at a convenient location.

4.2. Compiling and Installing Ruby

At this point you need to decide where on your file system you want to install the Ruby interpreter patched with caller_for_all_threads. Unless you are really into adrenaline rushes and extreme sports, it is not a good idea to override any existing interpreter installation. Instead, it is better to choose a completely new location, say /usr/local/ruby-1.8-threaddump.

  ./configure --prefix=/usr/local/ruby-1.8-threaddump
  make
  sudo make install

4.3. Setting Your Environment

You now have a fresh Ruby interpreter installed under /usr/local/ruby-1.8-threaddump. You just need to make sure that you are using this one (and not another installed on your system). The simplest way to achieve this is to set you environment accordingly:

  export PATH=/usr/local/ruby-1.8-threaddump/bin:${PATH}
  export LD_LIBRARY_PATH=/usr/local/ruby-1.8-threaddump/lib:${LD_LIBRARY_PATH}

You can then check that your environment is properly configured with:

  which ruby

which should return /usr/local/ruby-1.8-threaddump/bin/ruby

4.4. Installing RubyGems

Chances are you need a little bit more than a bare-bones Ruby interpreter, and soon you will need to install some RubyGems too!

So let’s get this out of the way by having a dedicated RubyGems install for this interpreter. Go back to the directory you typically use to build software or hack some code (for instance ~/projects) and execute the following commands:

  curl -OL http://de.mirror.rubyforge.org/rubygems/rubygems-1.3.1.tgz

  tar zxvf rubygems-1.3.1.tgz

  cd rubygems-1.3.1

  sudo ruby setup.rb --prefix=/usr/local/ruby-1.8-threaddump

Once the installation completes, you have a fully functioning gem install for your patched ruby interpreter.

Unfortunately RubyGems has a tendency to not install files under the correct directory. So please check first that you have a rubygems.rb file under /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8. If not, the gem command will complain with:

  `require': no such file to load -- rubygems (LoadError)

in which case you just need to manually copy the files to the correct location as indicated below:

  cd /usr/local/ruby-1.8-threaddump
  sudo cp -f  ./lib/rubygems.rb ./lib/ruby/site_ruby/1.8/
  sudo cp -f  ./lib/ubygems.rb  ./lib/ruby/site_ruby/1.8/
  sudo cp -rf ./lib/rubygems    ./lib/ruby/site_ruby/1.8/
  sudo cp -rf ./lib/rbconfig    ./lib/ruby/site_ruby/1.8/

You can double-check that everything is in the intended place with

  which gem

which should return /usr/local/ruby-1.8-threaddump/bin/gem. That’s it! You can then use the gem command in the usual way:

  gem list
  gem install rcov

Original web site design by: JFX diz*web.