Effortless Thread Dump for Ruby:

Dump the Stack Trace of All the Threads in your Ruby VM with caller_for_all_threads (M.R.I. 1.8)

Last update: July 22, 2008

1. Patching Your Ruby VM

To gather the stack trace for all threads you need to be able to switch between the different (green) thread contexts. Doing this requires quite a bit of “intimacy” with the Ruby interpreter and its setjmp / longjmp handling. This is why caller_for_all_threads cannot be implemented as a simple native gem – you will need to patch the Ruby interpreter first.

1.1. Patching M.R.I 1.8

Go in a directory that you typically use to build software or to hack some code (for instance ~/projects) and execute the following commands:

1.1.1. M.R.I 1.8.7

curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p22.tar.gz

tar zxvf ruby-1.8.7-p22.tar.gz

cd ruby-1.8.7-p22

curl -O http://xray.rubyforge.org/svn/patches_for_mri/↩
               caller_for_all_threads_patch_for_MRI_1.8.7.diff

patch -p0 < caller_for_all_threads_patch_for_MRI_1.8.7.diff

1.1.2. M.R.I 1.8.6

curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p114.tar.gz

tar zxvf ruby-1.8.6-p114.tar.gz

cd ruby-1.8.6-p114

curl -O http://xray.rubyforge.org/svn/patches_for_mri/↩
               caller_for_all_threads_patch_for_MRI_1.8.6.diff

patch -p0 < caller_for_all_threads_patch_for_MRI_1.8.6.diff

That’s it! Now is time to compile the Ruby interpreter and install it at a convenient location.

1.2. Compiling and Installing Ruby

At this point you need to decide where on your file system you want to install the Ruby interpreter patched with caller_for_all_threads. Unless you are really into adrenaline rushes and extreme sports, it is not a good idea to override any existing interpreter installation. Instead, it is better to choose a completely new location, say /usr/local/ruby-1.8-threaddump.

./configure --prefix=/usr/local/ruby-1.8-threaddump
make
sudo make install

1.3. Setting Your Environment

You now have a fresh Ruby interpreter installed under /usr/local/ruby-1.8-threaddump. You just need to make sure that you are using this one (and not another installed on your system). The simplest way to achieve this is to set you environment accordingly:

export PATH=/usr/local/ruby-1.8-threaddump/bin:${PATH}
export LD_LIBRARY_PATH=/usr/local/ruby-1.8-threaddump/lib:${LD_LIBRARY_PATH}

You can then check that your environment is properly configured with:

which ruby

which should return /usr/local/ruby-1.8-threaddump/bin/ruby

1.4. Installing RubyGems

Chances are you need a little bit more than a bare-bones Ruby interpreter, and soon you will need to install some RubyGems too!

So let’s get this out of the way by having a dedicated RubyGems install for this interpreter. Go back to the directory you typically use to build software or hack some code (for instance ~/projects) and execute the following commands:

curl -O http://rubyforge.rubyuser.de/rubygems/rubygems-1.1.1.tgz

tar zxvf rubygems-1.1.1.tgz

cd rubygems-1.1.1

sudo ruby setup.rb --prefix=/usr/local/ruby-1.8-threaddump

Once the installation completes, you have a fully functioning gem install for your patched ruby interpreter.

Unfortunately RubyGems has a tendency to not install files under the correct directory. So please check first that you have a rubygems.rb file under /usr/local/ruby-1.8-threaddump/lib/ruby/site_ruby/1.8. If not, the gem command will complain with:

`require': no such file to load -- rubygems (LoadError)

in which case you just need to manually copy the files to the correct location as indicated below:

cd /usr/local/ruby-1.8-threaddump
sudo cp -f  ./lib/rubygems.rb ./lib/ruby/site_ruby/1.8/
sudo cp -f  ./lib/ubygems.rb  ./lib/ruby/site_ruby/1.8/
sudo cp -rf ./lib/rubygems    ./lib/ruby/site_ruby/1.8/
sudo cp -rf ./lib/rbconfig    ./lib/ruby/site_ruby/1.8/

You can double-check that everything is in the intended place with

which gem

which should return /usr/local/ruby-1.8-threaddump/bin/gem. That’s it! You can then use the gem command in the usual way:

gem list
gem install rcov

2. Dumping the Stack Traces

Now that we have a patched Ruby VM, getting a thread dump is dead simple. Even a your grandma dude could do it!

2.1. It’s on Kernel, Stupid!

caller_for_all_threads is just a method on Kernel. So you could potentially call it from anywhere in your code.

We can convince ourselves with a little irb session:

$ ./irb 
irb(main):001:0> caller_for_all_threads
=> {#<Thread:0x22c6a8 run>=>["(irb):1:in `irb_binding'", 
                             "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52:in `irb_binding'", 
                             "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52"]}

irb(main):002:0> Thread.new { sleep 30; putc '.' }
=> #<Thread:0x2e4564 sleep>

irb(main):003:0> caller_for_all_threads
=> {#<Thread:0x2e4564 sleep>=>["(irb):2:in `irb_binding'", 
                               "(irb):2:in `initialize'", 
                               "(irb):2:in `new'", 
                               "(irb):2:in `irb_binding'", 
                               "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52:in `irb_binding'", 
                               "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52"], 
    #<Thread:0x22c6a8 run>=>["(irb):3:in `irb_binding'",
                             "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52:in `irb_binding'", 
                             "/tmp/foo/lib/ruby/1.8/irb/workspace.rb:52"]}
irb(main):004:0> 

Of course what you really want, is to write no code at all, just send a signal to the Ruby process and get a freshly baked stack trace.

2.2. Installing the Appropriate Signal Handler

The quickest way to get a nice thread dump signal handler is to install the XRay gem:

sudo gem install XRay

…and then, from your application code, require the file in charge of installing the thread dump signal handler:

require "rubygems"
require "xray/thread_dump_signal_handler"

In a Ruby on Rails application, you would typically require the thread dump signal handler in environment.rb or production.rb.

2.3. Triggering the Thread Dump

At this point you can trigger a full thread dump, at any time, just by sending the QUIT signal to your Ruby process:

kill -QUIT <pid of your ruby process>

Let’s put it all together. Create a simple Ruby script under /tmp/sample.rb:

#!/usr/bin/env ruby

require "rubygems"
require "xray/thread_dump_signal_handler"

a_thread =Thread.new do
  loop { sleep 1; puts "." }
end

a_thread.join

Launch it with the (patched) Ruby interpreter:

/usr/local/ruby-1.8-threaddump/bin/ruby /tmp/sample.rb

Then open another terminal window and discover the pid of your Ruby process. For instance do this with:

$ ps auxww | grep sample.rb | grep -v grep
 ph7       9956   0.0  0.1    76892   2836 s002  S+    6:19PM   0:00.04 ruby /tmp/sample.rb

It is time to send the QUIT signal to this process to get our thread dump:

kill -QUIT 9956

At this point a nice thread dump should be ready for you in the terminal you started sample.rb in:

=============== XRay - Thread Dump ===============

------------------------------------------------------------------------------
#<Thread:0x255c9c sleep> alive=true priority=0
------------------------------------------------------------------------------
    /tmp/sample.rb:7
      \_ /tmp/sample.rb:7:in `loop'
      \_ /tmp/sample.rb:7
      \_ /tmp/sample.rb:6:in `initialize'
      \_ /tmp/sample.rb:6:in `new'
      \_ /tmp/sample.rb:6

------------------------------------------------------------------------------
#<Thread:0x22c6a8 run> [main] [current] alive=true priority=0
------------------------------------------------------------------------------
    /tmp/foo/lib/ruby/gems/1.8/gems/XRay-1.0.3/lib/xray/thread_dump_signal_handler.rb:9
      \_ /tmp/sample.rb:10:in `call'
      \_ /tmp/sample.rb:10:in `join'
      \_ /tmp/sample.rb:10

=============== XRay - Done ===============

Of course you can get more than one thread dump, and keep sending QUIT signals to get multiple thread dumps if this is what you want!

kill -QUIT 9956
kill -QUIT 9956
kill -QUIT 9956

3. Conclusion

Hopefully you will find caller_for_all_threads useful for troubleshooting your production problems. I would love to hear from you and get your feedback: get in touch!