Jeremy Durham

Regardless of what Monty Python or Indiana Jones might have us believe, or not believe, there is no Holy Grail. In our case, the Holy Grail is the ability to run browser based Cucumber tests expediently. We’ve been chasing the Holy Grail for some time, and it ends today.

We started using Cucumber as we got more involved in the agile process. Unfortunately, like most other people, we were unable to catch our unicorn, and we never got our “customers” to write acceptance tests. Considering that we mainly do work for the government, I suppose we couldn’t expect to really have a lot of success getting them to write acceptance tests, but we subscribed to the process. The road was paved with good intentions.

How did it all go so wrong? As with many tools, we began to lean heavily on Cucumber. Our model, controller, and view specs started to be neglected, mainly as we spent serious amounts of time trying to get our javascript tests to run. Basically, very little of our applications will function without JavaScript.

The time we spent writing Cucumber features paled in comparison to the time we spent running Cucumber features. I previously blogged about my attempts to mitigate the issue. We now have another project that has hit 30 minutes per build. Obviously, we are in a unique situation and if you can keep your JavaScript tests down to <5 minutes, Cucumber + Selenium is probably fine and you should continue what you’re doing.

Recently, something snapped. Ben mentioned that his friend had rejected Cucumber and exclusively written view specs with graceful degradation. I respect the merits of doing that, but it can be difficult to sell the idea of building functionality in such a way that it will never be used, or specifically for testing purposes.

After a series of conversations, we’ve come up with a good direction. Our intention is to give more time to model and controller specs, use view specs or cucumber without selenium to test our views, and Jasmine to test our JavaScript. We’ll probably even mix in a bit more (GASP) manual testing, at least for the time being. With the time we save not writing javascript tests and running the full hour plus test suite, we’ll should have additional time to invest in researching other testing tools and strategies as well.

Ultimately, we’d love to increase our test coverage and validity, decrease our test running time, and as a by-product increase developer happiness. There’s nothing more demoralizing than finishing a feature, only to have to wait an hour or so to be able to push it and start on your next one. Ideally we’ll return to the days where running tests 50-100 times a day is not unrealistic. Right now, we’re lucky to run the full suite 5-6 times a day.

Cucumber, you’ve been good to us, and although you probably won’t be going away completely, it feels like the end of an era. Hopefully the next phase will bring faster builds and better tests. I’m happy we’re ending our quest for the Holy Grail, and although we’re not holding the Holy Grail in our hands, we’re more knowledgeable and ready to move on.