RSpec or Minitest, How Does One Choose?

A few months ago I started learning Ruby on Rails with two tutorials: Agile Web Development with Rails 4 and the Ruby on Rails Tutorial. The former used Minitest as its test framework, and the latter used RSpec. At the time I had some minor confusion as to why these two tutorials would use different test frameworks, but otherwise, I didn't think much of it. I was somewhat taken with RSpec's natural language approach to writing tests, and I figured I'd continue using RSpec.

Then a few weeks ago I read 7 Reasons Why I'm Sticking with Minitest and Fixtures in Rails by Brandon Hilkert, and I started reconsidering using RSpec, Factory Girl, and Capybara for Rails testing. Maybe RSpec was making things unnecessarily complicated without providing any real value with its closer-to-English tests. I decided to take a deeper look at the differences between RSpec and Minitest by converting one of the tutorials' tests to the other framework and doing a direct comparison of them.

Since I haven't had a lot of experience with RSpec yet and I'm not comfortable or familiar with all of its features, it was probably easier to convert the Ruby on Rails Tutorial RSpec specs to Minitest tests, so that's what I did. The complete sample app with both test suites is up on GitHub for your viewing pleasure. I tried to do as direct a conversion as possible, and I make no claims about being a testing guru, so there are most likely better ways to do the Minitest tests than the way I did them. I did learn a lot in the process, though. Here are some of those findings.

Setup for Minitest is Much Easier

Not that the setup for RSpec, Factory Girl, and Capybara is hard, but it is something that needs to be done and maintained if you're going to use these gems. You'll also most likely use Selenium Web Driver and Database Cleaner as well as other gems for an RSpec testing stack, so at minimum you've got five additional gems to keep track of and stay current on features, usage, and issues.

Minitest and fixtures are already built into Rails, so setup is trivial and it works out of the box. There are a few configuration parameters that you may want to tweak, but the setup is dramatically simplified. Simple is good; keep it simple.

Run Time for Minitest is not Substantially Faster

I was a bit surprised by this result. After running the test suite ten times for both RSpec and Minitest, I got average run times of 13.60s and 12.68s, respectively. That makes Minitest less than 10% faster for this test suite, which is relatively minor compared to how much speedup I would likely get from moving from my slow 5400rpm laptop hard drive to a fast SSD or spending some time optimizing the tests. Actually, I found that plugging in my laptop so the processor runs at top speed knocks 4 seconds off both run times. That's a 30% improvement for both test suites. At any rate, performance is probably not a good reason to pick one framework or the other. There are much better ways to improve test times with either one of them.

Code Size for Minitest is Significantly Smaller

As reported by rake stats, the lines of code (LOC) for the Minitest tests came to 506 LOC, and the specs totaled 701 LOC. With more careful consideration of thoroughly testing page elements only once, using tests that target unique page identifiers when testing the same page repeatedly, and refactoring duplication in tests, the Minitest LOC could be further reduced.

Why are tests so much smaller than specs? Let's look at a few examples. First, here's the spec in the user model for verifying that a user with the admin flag set is really an admin:

  describe "with admin attribute set to 'true'" do
    before do
      @user.save!
      @user.toggle!(:admin)
    end

    it { should be_admin }
  end

The corresponding Minitest code is:

  test "with admin attribute set to 'true'" do
    @user.save!
    @user.toggle!(:admin)
    assert @user.admin?
  end

They are very similar, but the spec needs a before block that requires a couple extra lines. This is a typical way that RSpec adds lines to specs while Minitest doesn't need these extra blocks. Moving on to a more complicated example, but still in the user model, here is a spec for verifying that a user can follow and stop following another user:

  describe "following" do
    let(:other_user) { FactoryGirl.create(:user) }
    before do
      @user.save
      @user.follow!(other_user)
    end

    it { should be_following(other_user) }
    its(:followed_users) { should include(other_user) }

    describe "followed user" do
      subject { other_user }
      its(:followers) { should include(@user) }
    end

    describe "and unfollowing" do
      before { @user.unfollow!(other_user) }

      it { should_not be_following(other_user) }
      its(:followed_users) { should_not include(other_user) }
    end
  end

Notice the use of Factory Girl to create a second user here, and using let to bind that user to a variable. In Minitest the same verification is much easier and more succinct:

  test "following" do
    other_user = users(:one)
    @user.save
    @user.follow!(other_user)

    assert @user.following?(other_user)
    assert @user.followed_users.include?(other_user)
    assert other_user.followers.include?(@user)

    @user.unfollow!(other_user)
    assert_not @user.following?(other_user)
    assert_not @user.followed_users.include?(other_user)
    assert_not other_user.followers.include?(@user)
  end

Minitest uses fixtures (with users(:one)) to accomplish the same binding to a variable, but it is much more direct. The flow of the test is also much cleaner and shorter without the extra describe and it blocks. It's a full 6 lines shorter than RSpec, not counting blank lines, and I think it's actually easier to read without all of the extra line noise.

Here's one last example from the user page integration tests that verifies the page that shows an index of all users. It's a longer test that signs in a user, tests that the contents of the user index page is correct, tests that pagination is present, and tests that there are delete links that work correctly when the signed in user is an admin.

  describe "index" do
    let(:user) { FactoryGirl.create(:user) }
    before(:each) do
      sign_in user
      visit users_path
    end

    it { should have_title('All users') }
    it { should have_content('All users') }

    describe "pagination" do
      before(:all) { 30.times { FactoryGirl.create(:user) } }
      after(:all) { User.delete_all }

      it { should have_selector('div.pagination') }

      it "should list each user" do
        User.paginate(page: 1).each do |user|
          expect(page).to have_selector('li', text: user.name)
        end
      end
    end

    describe "delete links" do
      it { should_not have_link('delete') }

      describe "as an admin user" do
        let(:admin) { FactoryGirl.create(:admin) }
        before do
          sign_in admin
          visit users_path
        end

        it { should have_link('delete', 
                              href: user_path(User.first)) }
        it "should be able to delete another user" do
          expect { click_link('delete', match: :first) }.to \
            change(User, :count).by(-1)
        end
        it { should_not have_link('delete', 
                                  href: user_path(admin)) }
      end
    end
  end

Factory Girl is used multiple times, and there is a fairly complex set of nested describe blocks making up this set of tests. There is a lot going on here, and the nesting makes things more difficult to read than it needs to be. Now look at the Minitest version:

  def setup
    @user = users(:one)
    @user.password = "foobar"
    sign_in @user
    get users_path
  end

  test "index" do
    assert_select 'title', full_title('All users')
    assert_select 'h1', 'All users'
  end
  
  test "pagination" do
    assert_select 'div.pagination'

    User.paginate(page: 1).each do |user|
      assert_select 'li', user.name
    end
  end

  test "delete links" do
    assert_select 'delete', false

    admin = users(:admin)
    admin.password = 'foobar'
    sign_in admin
    get users_path

    assert_select 'a[href=?]', 
                  user_path(User.first), 
                  'delete' 
    assert_difference 'User.count', -1 do
      delete user_path(User.first)
    end
    assert_select 'a[href=?]', 
                  user_path(admin), 
                  text: 'delete', 
                  count: 0
  end

I find this much easier to read and understand. The setup method is actually used for all of the tests in the user pages integration test file, so signing in the user and visiting the user index page is shared among all of the tests. There's a fixture that takes care of the user creation tasks, so all you see in the code is users(:one) and users(:admin) in place of the Factory Girl code for the RSpec version.

The tests are also split into the three main areas of focus - basic content, pagination, and delete links - in a much more straightforward way. The testing has a more direct feel to it and is much easier to follow than the RSpec version. I also really like the power of assert_select. Nearly all DOM verification can be done with this one assert method, and it's easy to learn and understand. It's also easy to wrap it in little helper methods to make new assert methods that are geared specifically for certain types of DOM tests, like assert_title or assert_link for verifying the presence of title or link text, respectively.

Minitest is Much Easier to Use

Sometimes a DSL isn't an advantage if you have to learn the DSL to do something that's not related to your product's domain. A test DSL that is built mostly for its own sake, not to make tests easier to write for the product you're testing, doesn't provide much benefit. A DSL that makes it easier to write the software for your product's domain provides the real benefits. A DSL that you have to take extra time to learn so that your tests read like natural language might be a waste of time.

RSpec is a DSL that risks focusing too much on making the perfect testing language and not enough on making a testing interface that is easy to understand and build on to efficiently test the Rails app that you really care about. Minitest is much closer to Ruby. It's a simple framework that provides a nice set of primitives that you can use to build out a test suite quickly and efficiently.

I'm sure a lot of people love writing tests in RSpec, and it feels very natural once you learn it well. That's a great thing, and I'm glad that it works for those people. But for me, RSpec seems to be an end in itself because it just wants to be a pretty testing language, whereas Minitest is a means to writing a clear, straightforward test suite that gets the job done. I want a testing framework that enables me to write tests quickly and get them out of the way so I can focus on improving the production code. Minitest does that for me, so that's the framework I'm going to use.