Jun 052016
 

If you’re a Ruby or Rails developer looking for some advice on how to get better at integration testing: congratulations! You’ve reached the highest level of difficulty in all of the areas of the stack you must conquer to become a great Ruby developer.

Integration testing is hard, but it doesn’t have to be. This the subtle of this truth lies in the fact that you must be skilled in both the backend and front-end of your app: you must understand factories and your Ruby objects, and if you have a Javascript-heavy app, the deep fundamentals of Javascript as well.

First things first, you will want to learn how to debug in both the front and back-ends. For the sake of this post, I’m going to assume you have learned a backend debugging tool like byebug. If not, try this tutorial now.

Second, you need to know that Capybara is a syntax for writing Ruby – “a DSL” – for telling a browser what to do. It can work against several of different browsers – Firefox, Chrome, and ‘headless’ browsers you can’t see. If you use a browser you can see, you get the neat effect of being able to view your results as they run, which can be fun (and you should do it) but may not work on a Continuous Testing / Continuous Integration platform.

A headless browser (of which I will discuss two: webkit and poltergeist) is complex to debug, and requires a command of all the parts of the stack.

Occasionally, you may write some Javascript code that will work in one browser and not another (you should learn to avoid this) – that’s why you can run Capybara with a single syntax against many different browsers.

The bad news is, in short, despite it being 2016 and Rails having been around for nearly 12 years none of the drivers is perfect.

Sometime ago I wrote about a neat little trick to view console messages while debugging Capybara webkit.

Driver’s name Browser The Bad The Good
Selenium Firefox Firefox doesn’t let you paste into the console
Chrome Chrome The Chrome debugging experience has some annoying gotchas. Don’t try to open the debugger while your spec is running, unless you pause on the back-end (for example, byebug). If you do pause on the Rails-side, you should be able to also fall into the debugger on the Chrome driver side too. If you do actually manage to open Developer Tools, you can reasonably debug your Javascript
webkit headless There are problems with PATCH requests when using this legacy headless driver. Take note that this PATCH problem was fixed in PhantomJS version 2.0. Webkit also requires you install QT on your system. Webkit lets you inspect status codes using driver.status_code and as mentioned in the post above, console messages too.
poltergeist headless If you app makes PATCH requests, note that poltergeist needs you to be running on Phantom JS 2.0 or higher to be able to process PATCH requests corruptly (when they aren’t, they come through on the server side as empty requests)
By default, anything that is sent from your Javascript as a console message makes your spec run fail (this can be turned off).
“The Worst Except For All the Others” (as Churchill said). This is the one I use primarily. Your console.log output is automatically ported from your Javascript into your test results.

Here’s a list of other notes of things to keep in mind.

  1. You should be using Capybara version 2.7.1 or higher. Earlier versions do not wait for all sessions to close before kicking off Database cleaner’s truncation. When truncation happens before all sessions are closed, bad things happen (like intermittent failing tests). Waiting and timing is explained in detail below.
  2. This applies to you if you app makes PATCH requests: Make sure you are on Phantom JS 2.0 or higher. Note this is a binary to install and on CI server it probably is a global (shell) configuration. (On ours, Semaphore, you need to specify the Phantom JS in the global build commands, not just in your Gemfile.) You to be running on Phantom JS 2.0 or higher to be able to process PATCH requests corruptly. When they aren’t, they come through on the server side as empty requests, which can lead to unexpected results.
  3. Capybara-webkit sucks. It just does. Don’t use it. The intermittent issues alone are enough to throw it out. Use Poltergeist instead. It was an older technology and by and large it has been replaced by Poltergeist. Experienced developers know this and don’t use webkit for this reason. Junior developers fight in vein trying to get webkit to work and waste lots of time believing in something that simply is a shitty piece of technology.
  4. When working with ChromeDriver note that it is annoyingly difficult to open the Developer Tools while the test is running. This is a knonw-issue, and the Chrome developers advise you pause your test to open Chrome Dev toos. This is explained here.
  5. When using Database cleaner with Truncation, Make sure you have it in an append_after hook and not in config.after(:each) (several tutorials will mistakenly lead you down the wrong path here.) It should look like this:
    config.append_after(:each) do
     DatabaseCleaner.clean
    end
  6. Prefer transaction instead of truncation for all non-Javascript tests (unit tests, controller tests, etc). For Javascript integration specs, you need truncation. An explanation about why can be found at https://github.com/DatabaseCleaner/database_cleaner#rspec-with-capybara-example
  7. Use Factories and don’t use fixture data. Fixture data can lead to brittle tests. Generally the entire Rails community has learned from the Dark Days and recommends factories over fixtures.
  8. Don’t use connection pooling. Some people on the internet will tell you to use connection pooling to solve thread-locking problems – don’t listen to them. Capybara already has dealt with this under the hood, make sure you are on a recent version of Capybara.
  9. Avoid using .trigger. Sometimes if an element isn’t visible Capybara will advise you when it fails you can ‘work around’ the element not being on the page by referencing the element and calling .trigger. You’re just trying to get around the on-screen-and-visible enforcement by Capybara, but this isn’t a good idea. If the thing isn’t on the screen and visible, it probably means there’s a bug and you want to catch that as a failure. Remember your tests are only as valuable as what they catch when things mess up.
  10. Circular Dependancy when trying to load ___
    This development issue that causes race condition (intermittent) failures has been explained on this Thoughtbot blog post.

    To fix if you’re on Rails 4.1 or prior, set allow_concurrency = false in test.rb (Rails 4.1 + earlier only)

    Set this in your config/environments/test.rb file set this:

    config.allow_concurrency = false

    You do not need this if you are on Rails 4.2 and above.

Timing

Timing is super hard to debug, but there’s an art to it. Tame your Capy specs like a lion tamer. Make them jump through hoops and bedazzel them to calm them down. You need to understand 3 things: Capybara’s native waiting, (2) a wait helper, and (3) an explicit sleep.

Capybara Native Waiting Behavior
If you’re on Capy 2.7 understand that Capybara natively waits for content on a page when you assert it to be there, even when Ajax and rendering might not have it ready to be there at the very moment the assertion runs. Thomas Wolpole, author of Capybara, advises me:

The way 2.7.1 is handling this is through middleware that keeps a counter of any current requests being processed by the app. First it tells the browser to visit about:blank and waits for that to happen, at which point the browser should not be initiating any more requests to the app. Then it waits for the active request counter to be 0, and then continues on.

Instead of using sleep, use expect(page).to have_content(…) to wait for the content you want to appear. Specifically I believe that using expect/have_content waits for the page to have the content you want it to have, but expect/value/to eq does not actually wait. For this reason, sprinkle in some expect(page).to have_content even when you don’t have to just to get Capybara to pause until the page is re-rendered.

You often find yourself writing

expect(page).to have_content(“xxx”)

over and over again. Contrary to the instinct to not Repeat Yourself, that’s a good thing! If this really irks you may write yourself helpers to make this repeated step more encapsulated. What you are really doing is putting the UX through it paces, so think of it like a player piano instructions not like the code you so work on to make beautiful.

It will be easier to do this if your app has natural-language responses like “You have logged in successfully.” For this reason you should encourage your Product Owners/Stakeholders to put in such natural language indicators – it makes your site easier, safer, and your regression suite more solid. And your users will appreciate instant feedback it too. If your product managers insist on ‘silent’ feedback, remember you can use Capybara to assert that things are or aren’t disabled, grayed-out, etc.

Basically, although you can sometimes get away with expectations that do direct Ruby object lookup, you really shouldn’t, or should use it as infrequently as possible

Wait Helpers (Ruby metaprograms Javascript)

Sometimes you’ll see a spec failure that will pass if you add sleep 1 or sleep 2. Avoid this, but use a very fast sleep (I use 0.1) when necessary. Instead of sleeping, turn off animations and write wait helpers for yourself to pause until certain conditions are met.

You should use wait helpers to wait for:

– Ajax requests that Capybara doesn’t seem to pick up natively (Later versions of Capy are supposed to count the number of outstanding Ajax requests but I’ve had difficulty getting this to work consistently. You can and should assert content is on page and prefer Capybara’s native waiting to a wait helper)

– Your app is doing something like initializing (you can even write your app to set itself a global flag when initialization has finished which can be checked from Capy helpers)

Here’s an example of a wait helper that waits for an Ajax request. Note here we are using page.evaluate_script to metaprogram Javascript by way of Ruby code, waiting until a condition is met before continuing the spec.

def wait_for_ajax
 counter = 0
 while page.evaluate_script(“$.active > 0”)
  counter += 1
  print “_”
  $stdout.flush
  sleep(0.1)
  if counter >= 100
   msg = “AJAX request took longer than 10 seconds.”
   if page.driver.respond_to?(:console_messages)
    msg << ” console messages at time of failure: ” + page.driver.console_messages.inspect
   end
   raise msg
  end
 end
end

Here’s an example of a wait helper that would wait for your app’s own initialization cycle, provided yourApp is the variable in Javascript where you app is namespaced, and when it is finished with its own initialization cycle it sets _initialized to true (on itself). You can write your own wait helpers appropriate to things you app does.

def wait_for_your_app
 counter = 0
 while page.evaluate_script(“typeof(yourApp) === ‘undefined’ || typeof(yourApp._initialized) === ‘undefined'”)
  counter += 1
  print “~”
  $stdout.flush
  sleep(0.1)
  raise “Your app failed to initialize after 10 seconds” if counter >= 100
 end
end

Explicit Sleeps

When all else fails sometimes you just need a sleep, which you just do in ruby as sleep X where [X] is the number of seconds you want to sleep.

You should use sleeps very rarely, but I’ve found they are needed in these cases:
– After an Ajax request, sometimes a sleep is needed to let the database catch up. (I try to keep these at about 0.5 seconds)
– A small timing delay (no longer than 0.1 seconds) for your app doing something like re-rendering

In theory you can wait for anything, so try to use Capybara’s internal waiting mechanisms first. In this order, your toolkit is:

1. Capybara’s internal waiting
2. A wait helper (as explained above)
3. An actual explicit sleep (try to keep all sleeps under 0.2 secs)

Remember each knife is shaper than the next, and so you should strive for minimal intrusiveness, but know that a combination of all three is likely necessary. The more astray you go from the art the more likely you experience timing delays.

Warning for anyone who has an expires_in set as a cache-control header in your controller endpoints (html or json).

Yes you! Go look in your code right now for expires_in set in your controllers and if you have any pay attention.

As I documented here, you’ve got to watch out if you have endpoints that have non-Zero cache-control headers on them. The headless driver (poltergeist or webkit) will hang onto the HTTP response between specs. This can be detrimental to you, if, say, the content of that endpoint’s response is what you are testing. In my case, I just used an inelegant hack to work around this- suggestions for improvements welcome.

if Rails.env.test?
 expires_in 0.minutes, :public => false
else
 expires_in 3.minutes, :public => true
end

Conclusion

Try to keep your feature specs to about 1-3 minutes to run per file, also maybe split them off when they are about 200-300 lines long. Be mindful of the total run time – since they are so valuable you can afford a little leeway here but keep in mind it slows down you time to develop new features.

Be careful about assertions that reach back into the database. Although you can get it to work, reloading objects and asserting things have changed is prone to race conditions, particularly with database cleaner. Remember that you have two threads operating separately, and even if you are able to do .reload on the object to get it into the right state, it’s actually nearly always better when writing Capybara specs to just assert the UX has changed the way you think it will.

And finally: Patience, discipline, know that others have been here before you and others will come here again. You are on the pinnacle of Rails development – don’t fall! Patience and faith.

Suggestion or feedback? Log in with your Stackoverflow, Github, Facebook or Google account to leave a comment.

May 252016
 
def wait_for_ajax
 counter = 0
 while page.evaluate_script(“typeof($) === ‘undefined'”)
  counter += 1
  print “^”
  $stdout.flush
  sleep(0.1)
  raise “Jquery not initialized after 10 seconds.” if counter >= 100
 end

 counter = 0
 while page.evaluate_script(“$.active > 0”)
  counter += 1
  print “_”
  $stdout.flush
  sleep(0.1)
  if counter >= 100
   msg = “AJAX request took longer than 10 seconds.”
   if page.driver.respond_to?(:console_messages)
    msg << ” console messages at time of failure: ” + page.driver.console_messages.inspect
   end
   raise msg
  end
 end
end

Feb 082015
 

First you need to know how to connect to your Heroku application using bash. You can then use the du -h command to get a read on how big the files inside your slug are.

heroku run bash -a yourappname

Once you’ve done that, here’s the magic command that will show you how big each folder in your slug is:

du -h --max-depth=1

Max-depth of 1 will give you 1 level deep output of how big each of those root folders are. You can change the max-depth to see more granular introspection of each folder. (However, realistically, it is probably more useful to cd down to some subfolders and run du -h --max-depth=1 again.)

Some observations I have made:

1. Obviously you basically want to get as much out of your assets/ folder as you possibly can. Almost everything I do these days is tied to some kind of managed content, so I can safely put almost all the images and videos in my app into a Paperclip-backed data model. Using S3, Paperclip stores all the different sized versions it will render remotely, so all that stuff stays out of the slug.

2. Beware of Gem bloat. If you don’t use it, you probably want to get it out of there.

3. Generally speaking just keep large files out of your app. If you’re needing a lot of larger data files re-think your architecture to work in a more distributed, background job kind of way that doesn’t require the files to be in the slug. (Remember, worker dynos can always download from or upload to S3 and storage on S3 is much cheaper than keeping things in your slug.)

4. Use .slugignore (well documented so I won’t go into it here.)

5. I found that using gems directly from Rubygems seems to save a lot of space compared to using gems which point to a Github repo.

You can recognize a gem pulled from its Rubygems version because it usually has few or no parameters, or only a version parameter:

(pulling from Rubygems)

gem ‘spree’, ‘2.1.12’

It turns out that because Spree can be a little lazy about building the Gems, they recommend you just point your Gemfile to the github repo. (To be fair, they maintain a lot of bugfixes across several branches, and pointing to a branch on Github makes much more sense if you have software that is retroactively fixed like Spree)

(Pointing to a github repo)

gem ‘spree’, git: ‘https://github.com/spree/spree.git’, branch: ‘2-1-stable’

The difference here is that Bundler can just pull a compiled gemfile in the first example, whereas it has to download the repo and turn it into a Gem in the second example. In my app, the latter (pulling from Rubygems) saves me a whopping 70 MB compressed in my slug. Although I haven’t confirmed, I suspect this is because Bundler is actually very efficient at packaging up a Gem and excludes all the support files you don’t want when you create the gem, whereas the ‘lazy’ way of just pointing to a Git repo has the side-effect of including all the files in the repo in your finished slug.

Oct 272014
 

Now with full test coverage using MiniTest.

Rubygems link:
http://rubygems.org/gems/nondestructive_migrations

Github Link:
https://github.com/jasonfb/nondestructive_migrations

A few things I learned here along the way:

MiniTest has not replaced TestUnit for the hard-core Rails developer. MiniTest worked great fro this, except Rails version 4.0.x, 3.1.x and 3.2.x needs “minitest-rails”, “~> 1.0”. I tried testing with Rspec but since Rails itself is tested with Minitest this was actually easier (since I needed access to some special tests written inside of Railtie)

I used a fantastic tool called Appraisal to run my tests suite against several different versions of Rails (3.1, 3.2, 4.0, 4.1, and 4.2). Getting this set up with a little bit of work, but well worth it. You can configure Travis CI to work seamlessly with Appraisal, so you can run your Gem against different versions of Ruby, Rails, and three different databases (SQLite, MySQL, and Postgres). See the .travis.yml and Appraisals inside the gem code for details.

When writing a Gem, you’ll want to explicitly include all your files in the files setting inside the Gemspec (example). This is important, and if you fail to do this those files will work when the Gem is loaded via git: or path: but not when it is pulled from RubyGems.

Rubygems forces you to publish explicit version numbers on your gems. You cannot overwrite an existing version number, but you are allowed to “yank” a version down to remove it from Rubygems.

 Posted by at 8:28 am  Tagged with:
Oct 212014
 

This is a useful little trick to get all the methods that you can call on any Ruby object. This is extremely helpful when poking around someone else’s code (like Gem code) that is poorly documented.

foo.methods

This will return a long list of methods that you can call on the object. That list will include all the methods on all the superclasses of the object, including the Object object methods (the grandaddy of all objects). That’s often not very useful, so you can filter out the superclasses’s methods and only look at the methods associated with the subclass using:

foo.methods(false)
 Posted by at 3:49 pm  Tagged with:
Sep 262014
 

Seriously why hasn’t anyone ever told me about this awesome tool?

With CapyCoder (below) you can literally record you interactions with code and the tool will write Capybara tests as you click.

I found that I couldn’t exactly move the recordings completely unedited into my capybara suite (for example, CapyCoder will create a visit path that goes to your dev site, which won’t work in a Capybara test suite), but with a little editing they worked splendidly.

Capycorder Chrome Extension

Aug 282014
 

This dredded error message happens very rarely when using rvm. Sometimes a command provided by a gem or any execution to a ruby script (or a rake task) results in this error on the command line:

dyld: Library not loaded

Typically I’ve seen this happen when I have recently re-sourced my .bash_profile. Basically all that has happened is that rvm has lost an important link. There are two easy non-invasive fixes and a 3rd invasive fix.

1. Run rvm list, then switch to any other ruby install, then switch back to the ruby install you want to run.

2. If that doesn’t work, try switching out of your current directory and back into it.

3. If that doesn’t work, you probably want to implode rvm and install it from scratch. (this can be done with rvm implode, but sometimes this hiccups and you must remove it manually from ~/.rvm)

Apr 222014
 

One of the often under appreciated situations in web development today are thing inherent to the process that create blockers inefficiencies when a developer is doing his or her day-to-day work. I am firm believer that inefficiencies are in fact blockers. and you should work hard to eliminate them. Although this might be difficult for ‘the business’ (managers, stakeholders) to understand, these things should be treated with the same importance as a “blocker” that you might bring up at stand-up. Failure to treat inefficiencies compound the problem, eventually leading to waste. Here are 9 things that make teams slow, and what you should do to address them on your team.

Continue reading »