Fixing order dependent tests in Minitest

Working on, a Rails 5 app which uses Minitest and Capybara, I ended up with some order dependent test failures that completely derailed things for a few days. In the end my brilliant son helped me to solve the problem and hopefully the key points will help someone else!

So, the app is a SaaS product and uses subdomains to separate tenant accounts. When the failure occurred, it was because the @current_account was unset in situations where it should have been and we ended up calling methods on nil. Those tests worked fine on their own, but during the whole test run, something was happening to make it mess up; some state or unsuspected dependency leaking from a previous test.

So I remembered Ryan Davis talking about this in his talk during RailsConf 2015, and how you could use minitest-bisect to do the leg work of trying to narrow down which other tests caused this dependency and failure. So, here were the steps I went through over the course of several days in order to get a test suite we could trust once again!

Step 1 – realise you have a problem!

This bit is easy. If your test suite runs green on one run, then throws up lots of red on a second when there have been no changes to the code, you have an order dependent test failure.

Step 2 – identify the test dependencies

Install minitest-bisect in your gem file in the development group and run your tests. Here’s the first gotcha, however. The readme and instructions are for a generic Ruby app; if you’re using Rails, you need to pass it the right file names to act on and finding the right incantation took a little googling!

minitest_bisect --seed 45514 -Itest $(find test -type f -name \*_test.rb)

Run this in terminal in the app directory, replacing the above seed with the value from your own unsuccessful test run.

This should take a few minutes to run, and will gradually narrow down the set of files and an order to run them in which generate the failure.

Eventually you should get something like

...many things...

Final reproduction:

Run options: --seed 45514 -n "/^(?:Create account Feature Test#(?:test_0001_Load the right home page)|Manage::JobsControllerTest#(?:test_should_create_new_job|test_should_get_index))$/"

Now you can pass the options displayed to the test runner and reproduce the subset of the problem.

Step 3- Identify the conditions which cause the failure

Steps 1 and 2 are, sadly, the easy bit. The tricky bit is then working out what is happening and what to do about it! But at least now you have a workable surface area to attack. In this case, just in case anyone else hits this slightly unusual edge case, our app uses the ActsAsTenant gem and requires a subdomain for particular areas to work properly, so we use a setup method with the host! method to set it before the tests that need this.

def setup
  host! ""

But the problem was that if prior to those test files running, a feature test was run that used “visit xxx_path”, the host! method in later tests no longer had any effect because the URL had already been set.

feature "View home page" do
  scenario "generic home page" do
    visit root_path
    page.must_have_content "Hireist"
  scenario "home page for account" do
    visit "http://verso.hireist.test"
    page.must_have_content "Verso"

Step 4 – Work out the dependency and fix it!

This is where I hit a brick wall, and my son, a talented Ruby developer, stepped in to help. It took some time but eventually using byebug and pry we were able to step through the code and dig down into the framework to try to find out where the domain was being set and why it leaked across test instances. We found that when a feature test runs, it set the default_url_options to “test.local” and this got set in a class variable – hence the persistence across test runs.

Because our integration test had the relevant classes in its ancestry chain, it turned out we could access the options hash inside our tests and reset the URL options before our test runs, so that the host! method would always have the desired effect.

In our test_helper.rb:

def clean_url_environment
  app.routes.default_url_options = {}

And in our setup method in the controller tests:

def setup
  host! ""

This was a tough one to fix, and it took sustained effort and more than one pair of eyes to do it. But without the ability to bisect the test runs (thanks to Ryan Davis), we’d probably never have gotten to the root cause.

How many boy scouts?

Sometimes we find ourselves working on stuff that is less than scintillating, maybe discussing the implementation details of yet another pointless cookie notification bar or whether it would be a bad thing to implement one of those things that highlights random words on your site like “gambling” and pops up ads if you accidentally mouse over them [yes! – Ed.]; at such times it’s sometimes tempting to dream you were instead working in Silicon Valley on some ground-breaking project like self-driving cars.

I have an antidote for that though; suppose you really were working on the software that guides a self-driving car. That would be awesome, right? Even the worst implementation of an AI-controlled vehicle is soon likely to be many times safer for the passengers and the world at large than even the best human driver, and there are really smart people working on the problems right now.

I’ve been reading this amazing essay from Paul Ford, “What is Code?”, and in it he explains why if those rockstar ninja developers exist, they’re probably going to have pretty high standards as to what problems they work on and are therefore probably not going to be sitting in the next cubicle in a typical dev team.

They’re not interviewing at your crappy company for your crappy job. They’re not going to come and rescue your website; they’re not going to make you an app that puts mustaches on photos; they’re not going to listen to you when you offer them the chance to build the next Facebook, because, if they exist, they are busy building the real Facebook.

Very astute, right? But it was this next sentence that made me glad I’m not one of them.

Sometimes they’re thinking about higher mathematics, or how to help a self-driving car manage the ethical choice between running over a squirrel and driving off a cliff. Or they’re riding their bikes, or getting really into pottery.

If you’re the one designing the anti-collision software, you’re the one who has to tell it how to decide in emergency situations. So the squirrel case is pretty easy (once you train it to reliably differentiate between a child in a furry hat and a squirrel, but that’s another matter) – bye bye Mr Squeaks, right?

But how would you suggest to your automaton that it handle the sort of situation that psychologists use to bother otherwise happy people; like, assuming that its brakes have failed and it’s hurtling down a hill, able to steer but not stop, and it has to choose the lesser of two evils.

Do you suggest it should stay on the road and aim, for instance, at the mother and a child in a pushchair on the pedestrian crossing (two people with their lives ahead of them), or steer into the bus queue of twenty older people instead?

But what if there are boy scouts in the bus queue too?

If you’re a rockstar programmer and your next question is, “how many boy scouts?” there may be a job for you in Mountain View. Tell them I suggested you give them a call.

Now, what colour would you like your cookie notification bar, Mr Smithers?

Unique filenames when uploading in Zend Framework

This is typical Zend Framework; needing a simple file upload as part of a Zend Form, I turned to the built in helpers and yes, it’s easy to upload a file with minimal additional code. So far so good.

But it defaults to uploading the file with the original filename and to overwriting existing files. I cannot remember a time when this was the behaviour I wanted from a file uploader; I always want to preserve existing files instead of overwriting them, and I usually want to give it some additional information in the filename or folder path – things like account or user IDs and timestamps for example, or just random hashes.

Eventually managed to find a way of customizing it that didn’t require three times the code saved by using Zend Framework in the first place, with grateful thanks to Dean Clatworthy:

$post = $request->getPost(); // This contains the POST params
        if ($request->isPost()) {
            if ($form->isValid($post)) {
                $upload = new Zend_File_Transfer_Adapter_Http();
                $filename = $upload->getFilename();
                $filename = basename($filename);
                $uniqueToken = md5(uniqid(mt_rand(), true));
                $filterRename = new Zend_Filter_File_Rename(array('target' => '/path/to/uploads/' . $uniqueToken.$filename, 'overwrite' => false));
                if (!$upload->receive()) {
                    $this->view->message = 'Error receiving the file';
                $this->view->message = 'Screenshot(s) successfully uploaded';

Unique filenames when uploading using Zend_Filter_File_Rename (Zend Framework).

Check for common misspellings for emails on signup forms

Kicksend/mailcheck · GitHub.

Came across this excellent jQuery plugin, which links into your signup forms and prompts users who mistakenly enter their email address if the mistake is a common one. For example, if they type foo@gmail.con instead of it doesn’t prevent it submitting, but prompts them “Did you mean Works for hotnail and so on too. Very nice; apparently they measured how effective it was and it did reduce bounced signup emails by a measurable %.

Zend Framework 2 – blech

It’s been a while since I paid any attention to what they were doing in ZF2; I’d been aware of the discussion and some of the plans way back, but last time I checked in there was nothing much to see yet unless you were involved directly with the effort.

So I stopped by today and was impressed to see that ZF2 is in production and there to download as 2.0.6.

I read through the new getting started tutorial and experienced a sinking feeling. I’ve always had this love/hate relationship with Zend Framework. Not so much with the software, actually, as the project as a whole. Since starting to use it back in 2007, I’ve found it badly documented, constantly shifting and often seeming to lack a coherent direction. Because it changed so much and relatively quickly in the early days (and had massive new features parachuted in without much explanation in later versions, like the Application object) it’s always been hard to know the best way to do things, particularly when you need to grab it quickly to use on a new project after working on an existing one for an extended period.

There is lots of good stuff in ZF2 – I’m really pleased to see unit testing front and centre in the getting started, for example, and I like the new more convenient form rendering methods.

But I’m shocked at how lengthy and verbose the getting started tutorial is, just to get to the point where you’re printing out some records on a page. I’m not saying ZF should have scaffolding or that scaffolding is actually particularly useful to have; on the other hand, I ask myself, do I really have to type all that in, especially when other frameworks are so much more expressive? Or more realistically, copy/paste it all every time you need it. The routing code is amazingly ugly with an extraordinarily low signal to noise ratio – just imagine how much boilerplate you’ll be scrolling through in a real app just to read the dozen or so bits of actual information in that.

This is not a technical criticism, incidentally, it’s an aesthetic and usability thing. I’m not disputing the architectural decisions; the guys who write and maintain ZF are way above me as programmers. But as an average, reasonably experienced web developer, I want to write code that I like to look at, code that is “user-friendly” for the developers in the same way that the UI should be user-friendly for the end users.

Zend Framework.

Good signup form


Thought this signup form from Kickstarter was rather well done – presenting the signup form in its entirety on the same page as the login form; they obviously know that a large number of users each day are new to the service so give them as few opportunities as possible to fall by the wayside. Interesting how services like Twitter and Facebook are becoming standards for authentication; it’s as though OpenID has finally come of age, all we needed were central services with sufficient reach to act as the broker of your credentials. When I first signed up to Stackoverflow back at the start of the public beta, there were few available services I would willingly choose to trust as my provider.

Interesting how the visual focus of the page is heavily weighted toward creating a new account. I didn’t even get as far as reading the Facebook option before I started typing. I wonder if that’s intentional.