Fixing order dependent tests in Minitest

Working on Hireist.com, a Rails 5 app which uses Minitest and Capybara, I ended up with some order dependent test failures that completely derailed things for a few days. In the end my brilliant son helped me to solve the problem and hopefully the key points will help someone else!

So, the app is a SaaS product and uses subdomains to separate tenant accounts. When the failure occurred, it was because the @current_account was unset in situations where it should have been and we ended up calling methods on nil. Those tests worked fine on their own, but during the whole test run, something was happening to make it mess up; some state or unsuspected dependency leaking from a previous test.

So I remembered Ryan Davis talking about this in his talk during RailsConf 2015, and how you could use minitest-bisect to do the leg work of trying to narrow down which other tests caused this dependency and failure. So, here were the steps I went through over the course of several days in order to get a test suite we could trust once again!

Step 1 – realise you have a problem!

This bit is easy. If your test suite runs green on one run, then throws up lots of red on a second when there have been no changes to the code, you have an order dependent test failure.

Step 2 – identify the test dependencies

Install minitest-bisect in your gem file in the development group and run your tests. Here’s the first gotcha, however. The readme and instructions are for a generic Ruby app; if you’re using Rails, you need to pass it the right file names to act on and finding the right incantation took a little googling!

minitest_bisect --seed 45514 -Itest $(find test -type f -name \*_test.rb)

Run this in terminal in the app directory, replacing the above seed with the value from your own unsuccessful test run.

This should take a few minutes to run, and will gradually narrow down the set of files and an order to run them in which generate the failure.

Eventually you should get something like

...many things...

Final reproduction:

Run options: --seed 45514 -n "/^(?:Create account Feature Test#(?:test_0001_Load the right home page)|Manage::JobsControllerTest#(?:test_should_create_new_job|test_should_get_index))$/"

Now you can pass the options displayed to the test runner and reproduce the subset of the problem.

Step 3- Identify the conditions which cause the failure

Steps 1 and 2 are, sadly, the easy bit. The tricky bit is then working out what is happening and what to do about it! But at least now you have a workable surface area to attack. In this case, just in case anyone else hits this slightly unusual edge case, our app uses the ActsAsTenant gem and requires a subdomain for particular areas to work properly, so we use a setup method with the host! method to set it before the tests that need this.

def setup
  host! "verso.example.com"
end

But the problem was that if prior to those test files running, a feature test was run that used “visit xxx_path”, the host! method in later tests no longer had any effect because the URL had already been set.

feature "View home page" do
  scenario "generic home page" do
    visit root_path
    page.must_have_content "Hireist"
  end
  scenario "home page for account" do
    visit "http://verso.hireist.test"
    page.must_have_content "Verso"
  end
end

Step 4 – Work out the dependency and fix it!

This is where I hit a brick wall, and my son, a talented Ruby developer, stepped in to help. It took some time but eventually using byebug and pry we were able to step through the code and dig down into the framework to try to find out where the domain was being set and why it leaked across test instances. We found that when a feature test runs, it set the default_url_options to “test.local” and this got set in a class variable – hence the persistence across test runs.

Because our integration test had the relevant classes in its ancestry chain, it turned out we could access the options hash inside our tests and reset the URL options before our test runs, so that the host! method would always have the desired effect.

In our test_helper.rb:

def clean_url_environment
  app.routes.default_url_options = {}
end

And in our setup method in the controller tests:

def setup
  clean_url_environment
  host! "verso.example.com"
end

This was a tough one to fix, and it took sustained effort and more than one pair of eyes to do it. But without the ability to bisect the test runs (thanks to Ryan Davis), we’d probably never have gotten to the root cause.