Assert Yourself: A Detailed Minitest Tutorial

This post introduces automated testing with Ruby’s Minitest software, with an emphasis on using assertions. It discusses basic test scaffolding; assertions and refutations; skipping and flunking tests; setup and teardown.

Note: This article was originally published on the Launch School blog on 2016–08–28

One of the biggest keys to producing quality software is properly testing your program under a wide variety of conditions. Doing this manually is tedious and error-prone, and frequently doesn’t get done at all. This leads to buggy software, and sometimes software that doesn’t work at all.

Fortunately, modern programming environments support automated tests. Such tests can be run repeatedly with minimal effort: it’s even possible to run them automatically prior to every commit or release. While automated testing doesn’t guarantee that your code will be bug free — the tests are only as good you make them — it can certainly reduce the number of bugs your users will encounter. Automated testing can also help you find and fix bugs earlier in the development cycle, and prevent a lot of needless debugging trying to track down a particularly nasty bug.

In this post, we will talk about Minitest, the standard software testing framework provided with Ruby. It isn’t the only software testing framework available, but being supplied automatically with Ruby is a major advantage.

In particular, we will discuss how to use Minitest assertions to test your software. We aren’t interested so much in the theory of automated testing, but in how to use Minitest.

Definitions

A testing framework is software that provides a way to test each of the components of an application. These can be methods or entire programs; the framework should be able to provide appropriate inputs, check return values, examine outputs, and even determine if errors occur when they should.

Testing frameworks provide 3 basic features:

• a way to describe the tests you want to run,
• a way to execute those tests,
• a way to report the results of those tests.

There is a hierarchy to tests. Since there isn’t any formal agreement on exactly which terms are used to describe this hierarchy, we will use the following definitions:

• A test step, or simply a test, is the most basic level of testing. A test step simply verifies that a certain expectation has been satisfied. For example, in a to-do application, you may want to verify that a newly created to-do is not yet marked as completed. Test steps employ either an assertion or an expectation depending on your testing framework.
• A test case is a set of actions that need to be tested combined with any appropriate test steps. For example, a test case for the above test step may include creation of the to-do object, a call to the #completed? method on that object, and, finally, an assertion that the return value of the #completed? method is false. Typically, only one test step is used per test case; some developers insist on this, others are more flexible. For brevity, some of our test cases may show multple test steps.
• A test suite is a collection of one or more test cases that, taken together, demonstrate whether a particular application facet is operating correctly. We use this term quite loosely: a test suite can test an entire class, a subset of a class, or a combination of classes, all the way up to the complete application. The test suite may be entirely in one file, or it may be spread out over several files.

There are other terms you will encounter in the testing world, but for our purposes, these 3 terms will be sufficient.

What is Minitest?

Minitest is a testing framework that comes with every standard Ruby distribution. It’s not quite as powerful and flexible as its cousin RSpec, but for most cases, Minitest will do everything you need. It provides multiple interfaces — assertions and expectations — and a choice of several different output formats.

In addition to being a testing framework, Minitest provides the ability to create and use mock objects and stubs, and the ability to run benchmarks. These topics won’t be discussed in this post, but may appear in a future post.

Assertions or Expectations?

As mentioned above, Minitest provides assertion-based and expectation-based interfaces.

In the assertions-based interface, the test writer supplies one or more classes that represent test suites. Within each test suite are one or more methods that define test cases. Finally, each test case method has some code that exercises some aspect of the item under test and then runs one or more test steps to verify the results. Here’s a simple example of a test suite for testing a square_root method:

require 'minitest/autorun'
class TestSquareRoot < Minitest::Test
  def test_with_a_perfect_square
    assert_equal 3, square_root(9)
  end
  def test_with_zero
    assert_equal 0, square_root(0)
  end
  def test_with_non_perfect_square
    assert_in_delta 1.4142, square_root(2)
  end
  def test_with_negative_number
    assert_raises(Math::DomainError) { square_root(-3) }
  end
end

In the expectations-based interface, the test writer uses a domain-specific language, or DSL, to describe the tests:

require 'minitest/autorun'
describe 'square_root test case' do
  it 'works with perfect squares' do
    square_root(9).must_equal 3
  end
  it 'returns 0 as the square root of 0' do
    square_root(0).must_equal 0
  end
  it 'works with non-perfect squares' do
    square_root(2).must_be_close_to 1.4142
  end
  it 'raises an exception for negative numbers' do
    proc { square_root(-3) }.must_raise Math::DomainError
  end
end

As you can see, assertions are little more than basic Ruby code: all you need to know is what modules to require; how to name your classes (test suites) and methods (test cases); and the methods (assertions) you need to call to perform each test. On the other hand, expectations have a DSL that needs to be learned, with commands like describe and it, and those odd must_* methods being applied to the return values of square_root. The DSL is more English-like, but you still need to learn the DSL even if you know English.

For the remainder of this post, we will concentrate on using Minitest assertions.

Writing a Simple Test Suite

Setting up a Minitest test suite isn’t difficult, but it also isn’t obvious when first starting out with Minitest. Before you start writing your first test suite, you need to know how to prepare it.

Typically, test suites are stored in in a special tests directory beneath your main application's development directory. For example, if you are working on a to-do application that is stored in /Users/me/todo, then you will place your test suite files in /Users/me/todo/tests. This isn't a requirement, but is good practice for source organization, particularly when working with large projects.

There are no universal conventions regarding how to name your test suites. In the absence of any rules imposed by projects guidelines, we recommend establishing your own naming conventions and sticking to them; for instance, a common convention is to name a test suite for a class named ToDo as t_to_do.rb or to_do_test.rb.

Once you know where your tests suites will live and how they will be named, you need to set up some scaffolding code for the tests. Your scaffolding code will look something like this:

require 'minitest/autorun'
require_relative '../lib/xyzzy'
class XyzzyTest < Minitest::Test
  def test_the_answer_is_42
    xyzzy = Xyzzy.new
    assert(xyzzy.the_answer == 42, 'the_answer did not return 42')
  end
  def test_whats_up_returns_doc
    xyzzy = Xyzzy.new
    assert(xyzzy.whats_up == "Doc", 'whats_up did not return "Doc"')
  end
end

We start by requiring minitest/autorun; this package includes everything you need to run most basic tests, and it ensures that all of the tests in your test suite will be run automatically when you run the test suite.

Next we require the file(s) that contains the code we want to test. You will usually need to use require_relative, and will need to figure out the relative name of the module with respect to the test suite file. In this example, the source code is found in ../lib with respect to the location of the test file.

We then define a class that inherits from Minitest::Test (or MiniTest::Test; you may use Minitest and MiniTest interchangeably). This class defines the test suite -- a collection of one or more test cases. The name of this class is not important; however, it is common to append or prepend Test to the name, and the rest of the name is often the name of the class or module being tested. What's important to Minitest is that the class inherits from Minitest::Test; Minitest runs tests in every class that inherits from Minitest::Test.

Finally, we have provided scaffolding for two example test cases: one that tests whether Xyzzy#the_answer returns 42, and one that tests whether Xyzzy#whats_up returns "Doc". Both test cases have a bit of code to set up the test (we create an Xyzzy object in both cases), and a test step that verifies each of the methods returns the expected answer. (We will return to the assert calls in a bit -- for now, just understand that each of these tests succeeds if the specified condition is true. If the condition is false, the test fails, and the message string specified as the second argument is printed as part of the failure message.)

Note that each of the test cases is represented by a method whose name begins with test_; this is required. Minitest looks for and runs all methods in the test suite whose name begins with test_. The rest of the method name is usually a short description of what we are testing; as you'll see later, well-named test cases will help make it easier to understand failures caught by Minitest.

Every assertion-based Mintest test suite you set up will look something like this. Some test suites may have multiple test suite classes; Minitest will run all of the test suite classes defined in the file. Some test suite classes will have only one or two test cases; others may have many test cases.

NOTE: If the code you are testing includes code that starts your app, see the section on testing startup code near the end of this post.

Writing Tests

While we won’t be doing any actual development in this post, it’s important to understand how testing fits into the software development cycle. Ideally, your test cases should be run before writing any code. This is frequently called Test-Driven Development (TDD), and follows a simple pattern:

1. Create a test that fails.
2. Write just enough code to implement the change or new feature.
3. Refactor and improve things, then repeat tests.

This is often called Red-Green-Refactor. Red describes the failure step; green describes the getting things working; and, of course, refactor covers refactoring and improving things.

Once you’ve been through these steps, you can move on the next feature and repeat all of the above steps.

This post is not about TDD. However, it is useful to use the TDD approach in developing tests at the lowest levels.

Example

Lets look at an example. Suppose we want to develop a method to calculate square roots rounded to the nearest integer. If we attempt to take the square root of a negative number, the method should return nil.

For brevity, we will write our square_root method in the same file as our tests -- this is not usually good practice.

First Test Case: square_root(9) should return 3

For our first test, we will take a simple case — most people know that the square root of 9 is 3, so let’s test that.

require 'minitest/autorun'
def square_root(value)
end
class SquareRootTest < Minitest::Test
  def test_that_square_root_of_9_is_3
    result = square_root(9)
    assert_equal 3, result
  end
end

This is our first test — it uses #assert_equal to assert that the result of square_root(9) is 3. (We will discuss #assert_equal in more detail later.)

We have also supplied a dummy square_root method just so the test case has something it can test. Note that we used the name square_root because our test case uses that name -- the test case is already driving our development by describing the interface we want to use.

If we run this file, we get the following results:

Run options: --seed 51859
# Running:
F
Finished in 0.000886s, 1128.2034 runs/s, 1128.2034 assertions/s.
  1) Failure:
SquareRootTest#test_that_square_root_of_9_is_3 [x.rb:9]:
Expected: 3
  Actual: nil
1 runs, 1 assertions, 1 failures, 0 errors, 0 skips

We can see that there is a failure, and it occurred on line 9 where #assert_equal is called. Now that we have something that fails, let's fix it:

require 'minitest/autorun'
def square_root(value)
  Math.sqrt(value)
end
class SquareRootTest < Minitest::Test
  def test_that_square_root_of_9_is_3
    result = square_root(9)
    assert_equal 3, result
  end
end

This time, we get:

Run options: --seed 42248
# Running:
.
Finished in 0.000804s, 1243.5305 runs/s, 1243.5305 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

Notice the . just below # Running:; we should see one . for every successful test case. If we see F's, we know that one or more test cases failed. If we see E's, we know that one or more test cases were completely broken. You may also see some S's here: an S indicates that a test case was skipped.

Second Test Case: square_root(17) should return 4

Our goal is to round the square root to the nearest integer, so the square_root of 17 should be rounded to 4. Lets write that test case and test it:

require 'minitest/autorun'
def square_root(value)
  Math.sqrt(value)
end
class SquareRootTest < Minitest::Test
  def test_that_square_root_of_9_is_3
    result = square_root(9)
    assert_equal 3, result
  end
  def test_that_square_root_of_17_is_4
    result = square_root(17)
    assert_equal 4, result
  end
end
1) Failure:
SquareRootTest#test_that_square_root_of_17_is_4 [x.rb:15]:
Expected: 4
  Actual: 4.123105625617661

We get our expected failure — we seem to be on the right track. The problem here is that we aren’t rounding the result, but are returning a Float value that closely approximates the square root of 17. Let's fix that:

require 'minitest/autorun'
def square_root(value)
  Math.sqrt(value).to_i
end
class SquareRootTest < Minitest::Test
  def test_that_square_root_of_9_is_3
    result = square_root(9)
    assert_equal 3, result
  end
  def test_that_square_root_of_17_is_4
    result = square_root(17)
    assert_equal 4, result
  end
end

This time, both tests pass.

Third Test Case: square_root(24) should return 5

The square root of 24 is approximately 4.9 which, rounded to the nearest integer, is 5. Lets write another test case:

require 'minitest/autorun'
def square_root(value)
  Math.sqrt(value).to_i
end
class SquareRootTest < Minitest::Test
  def test_that_square_root_of_9_is_3
    result = square_root(9)
    assert_equal 3, result
  end
  def test_that_square_root_of_17_is_4
    result = square_root(17)
    assert_equal 4, result
  end
  def test_that_square_root_of_24_is_5
    result = square_root(24)
    assert_equal 5, result
  end
end

This fails with:

1) Failure:
SquareRootTest#test_that_square_root_of_24_is_5 [x.rb:20]:
Expected: 5
  Actual: 4

We get our expected failure. The problem here is that we aren’t rounding the result, but are simply truncating it with #to_i. To round it, we need to use the #round method:

require 'minitest/autorun'
def square_root(value)
  Math.sqrt(value).round
end
class SquareRootTest < Minitest::Test
  def test_that_square_root_of_9_is_3
    result = square_root(9)
    assert_equal 3, result
  end
  def test_that_square_root_of_17_is_4
    result = square_root(17)
    assert_equal 4, result
  end
  def test_that_square_root_of_24_is_5
    result = square_root(24)
    assert_equal 5, result
  end
end

Success once again.

Fouth Test Case: square_root(-1) should return nil

In the real domain, square roots aren’t defined for negative numbers, and our requirements require that #square_root should return nil if it is passed a negative number. Once again, let's write a test case, this time using #assert_nil:

require 'minitest/autorun'
def square_root(value)
  Math.sqrt(value).to_i
end
class SquareRootTest < Minitest::Test
  def test_that_square_root_of_9_is_3
    result = square_root(9)
    assert_equal 3, result
  end
  def test_that_square_root_of_17_is_4
    result = square_root(17)
    assert_equal 4, result
  end
  def test_that_square_root_of_24_is_5
    result = square_root(24)
    assert_equal 5, result
  end
  def test_that_square_root_of_negative_number_is_nil
    result = square_root(-1)
    assert_nil result
  end
end

This fails with an error:

SquareRootTest#test_that_square_root_of_negative_number_is_nil:
Math::DomainError: Numerical argument is out of domain - "sqrt"
    x.rb:4:in `sqrt'
    x.rb:4:in `square_root'
    x.rb:24:in `test_that_square_root_of_negative_number_is_nil'

Note in particular that this problem is occurring in the call to square_root, not in the assertion. This means we have to do something in square_root to deal with negative inputs:

require 'minitest/autorun'
def square_root(value)
  return -5 if value < 0
  Math.sqrt(value).round
end
class SquareRootTest < Minitest::Test
  def test_that_square_root_of_9_is_3
    result = square_root(9)
    assert_equal 3, result
  end
  def test_that_square_root_of_17_is_4
    result = square_root(17)
    assert_equal 4, result
  end
  def test_that_square_root_of_24_is_5
    result = square_root(24)
    assert_equal 5, result
  end
  def test_that_square_root_of_negative_number_is_nil
    result = square_root(-1)
    assert_nil result
  end
end

Note that we are now returning -5 if the value is negative; this isn't the correct value for square_root, but we need to have a failing assertion before we can do a proper test. Returning a value we know to be invalid is one way to handle this.

With this change, we now get the failure we want:

1) Failure:
SquareRootTest#test_that_square_root_of_negative_number_is_nil [x.rb:26]:
Expected -5 to be nil.

without introducing any new failures or errors. We can now modify #square_root to return the correct value:

require 'minitest/autorun'
def square_root(value)
  return nil if value < 0
  Math.sqrt(value).round
end
class SquareRootTest < Minitest::Test
  def test_that_square_root_of_9_is_3
    result = square_root(9)
    assert_equal 3, result
  end
  def test_that_square_root_of_17_is_4
    result = square_root(17)
    assert_equal 4, result
  end
  def test_that_square_root_of_24_is_5
    result = square_root(24)
    assert_equal 5, result
  end
  def test_that_square_root_of_negative_number_is_nil
    result = square_root(-1)
    assert_nil result
  end
end

Success once again.

At this point, we can now refactor things to improve the code if we need to do so. After each refactor, we should rerun the tests to ensure that our most recent changes have not broken anything.

Test Sequence

One thing to keep in mind with files that contain multiple test cases (or multiple test suites) is that Minitest varies the order in which it runs the tests. In the example in the previous section for example, there are 4 tests, but the order in which they are run is completely at random. On one run, the 3rd, 1st, 4th, and 2nd tests may be run in that sequence, but on another run, the 2nd, 4th, 3rd, and 1st tests may be run.

This random sequence is intentional, and should be left as-is. Your tests should not be order-dependent; they should be written so that any test can be performed in any order.

If you have a sporadic issue that only arises when tests are called in a specific order, you can use the --seed option to run some tests in a known order. For instance, suppose you run some tests and get the following output:

Run options: --seed 51859
...
  1) Failure:
XxxxxTest#test_... [x.rb:9]:
Expected: 3
  Actual: nil
3 runs, 3 assertions, 1 failures, 0 errors, 0 skips

but a subsequent run produces:

Run options: --seed 23783
...
3 runs, 3 assertions, 0 failures, 0 errors, 0 skips

you can rerun the first set of tests by using --seed 51859 on the command:

$ ruby test/tests.rb --seed 51859

If you absolutely, positively need to always execute things in order, you must name all your test methods in alphabetical order, and include the command:

i_suck_and_my_tests_are_order_dependent!

As you might surmise, the developers of Minitest feel very strongly about this. Don’t do it!

Simple Assertions

As we saw above, tests are written using a wide variety of different assertions. Each assertion is a call to a method whose name begins with assert. Assertions test whether a given condition is true; refutations test whether a given condition is false. For now, we'll limit the discussion to assertions -- refutations, the negation of assertions, will be described later.

assert

The simplest possible assertion is the #assert method. It merely tests whether the first argument is "truthy":

assert(reverse('abc') == 'cba')

If the condition is truthy, the test passes. However, if the condition is false or nil, the test fails, and Minitest issues a failure message that says:

Expected false to be truthy.

which isn’t particularly helpful. For this reason, assert is usually called with an optional failure message:

assert(reverse('abc') == 'cba', "reverse('abc') did not return 'cba'")

which is slighty more informative. (As we’ll see in the next session, we can do even better than this.)

All refutations and almost all assertions allow an optional message as the final argument passed to the method. However, in most cases, the default message for methods other than #assert and #refute is good enough to not require a specific message.

The following assertions do not accept a message argument:

• assert_mock
• assert_raises
• assert_silent

Typically, #assert is used with methods that return true or false:

assert(list.empty?, 'The list is not empty as expected.')

More often, #assert simply isn't used; instead, we use #assert_equal with an explicit true or false value.

assert_equal

The most frequently used assertion is #assert_equal -- it tests whether an actual value produced during a test is equal to an expected value. For instance:

assert_equal('cba', reverse('abc'))

In this call, the first argument represents an expected value, while the second argument is the actual value — in this case, the actual value is the return value of reverse('abc').

#assert_equal uses the == operator to perform its comparisons, so you can compare complex objects if an #== method has been defined. If an explicit #== method is not defined, the default BasicObject#== is used, which only returns true if the two objects are the same object.

If the expected and actual values are equal, the test passes. However, if they are not equal, the test fails, and Minitest issues a failure message that says:

Expected: "cba"
  Actual: "some other value"

Since the == operator is used, you must be careful when asserting that a value is explicitly true or false:

assert_equal(true, xyzzy.method1)
assert_equal(false, xyzzy.method2)

The first of these assertions passes only if the return value of xyzzy.method1 is exactly equal to true; it doesn't check for truthiness like #assert. Similarly, the second assertion passes only if xyzzy.method2 returns false exactly; it fails if xyzzy.method2 returns nil.

assert_in_delta

If you’ve spent any time working with floating point numbers, you’ve no doubt encountered the strange fact that floating point numbers are not usually stored as exact representations; small precision inaccuracies occur in the least significant digits of some floating point numbers. Thus, something like:

value_a = 2
value_b = 1.9999999999999999
puts value_a == value_b

prints true even though the two values look like they should differ. This inaccuracy means that it is inadvisable to make comparisions for exact values in your program. The same holds true for your test cases: you should not assert that a floating point number has a specific value.

Instead, you should assert that a floating point number is near some value: how near is up to you. This is accomplished with the #assert_in_delta method:

assert_in_delta 3.1415, Math::PI, 0.0001
assert_in_delta 3.1415, Math::PI, 0.00001

In these two tests, the 3rd argument is the “delta” value: if the expected and actual answers differ by more than the delta value, the assertion fails. If the 3rd argument is not supplied to #assert_in_delta, a delta value of 0.001 is assumed.

In this example, the first test succeeds because 3.1415 differs from the actual value returned by Math::PI (3.141592653589793) by less than 0.0001 (the actual difference is approximately 0.0000027), while the second test fails because 3.1415 differs from Math::PI by more than 0.00001.

assert_same

The #assert_same method checks whether two object arguments represent the exact same object. This is most useful when you need to verify that a method returns the same object it was passed. For instance:

assert_same(ary, ary.sort!)

As with #assert_equal, the first argument is the expected value; the second argument is the actual value.

Note that equality of objects is not the same thing as being the same object; two objects can be different objects, but can have equivalent state. Ruby uses the #== method to determine equality, but BasicObject#equal? to determine that two objects are the same. #assert_same, thus, uses the #equal? method; since #equal? should never be overridden, you can usually be assured that #assert_same uses BasicObject#equal?.

Note, too, that BasicObject#== does the same thing as BasicObject#equal?, but most objects override #==, while no class should override #equal?. In most cases, you can assume that #assert_equal tests for equivalent states, while #assert_same tests for the same object.

assert_nil

#assert_nil checks whether an object is nil. This is most useful in conjunction with methods that return nil as a "no results" result.

assert_nil(find_todos_list('Groceries'))

assert_empty

#assert_empty checks whether an object returns true when #empty? is called on the object. If the object does not respond to #empty? or it returns a value other than true, the test fails. #assert_empty is most useful with collections and Strings.

list = []
assert_empty(list)

Note in particular that the assertion is named #assert_empty without a ?, but it checks the #empty? method.

assert_includes

#assert_includes checks whether a collection includes a specific object. If the collection does not respond to #include? or if the method returns a value other other than true, the test fails.

list = %w(abc def xyz)
assert_includes(list, 'xyz')

Note that the assertion is named #assert_includes with an s but no ?, but it checks the #include? method which has a ? but no s. Note also that this method departs from the "expected, actual" or "value" format used by some other assertions.

assert_match

#assert_match is used when working with String objects: often, you don't need to test an exact value, but just need to determine whether a String matches a given pattern. #assert_match uses regular expressions (regex) for those patterns.

Most often, assert_match is used with text whose content only needs to contain a few specific words, such as an error message:

assert_match(/not found/, error_message)

Setup and Teardown

Many test suites have some code that needs to be run either before or after each test. For example, a test suite that validates a database query may need to establish the database connection prior to each test, and then shut down the connection after each test. This can be easily accomplished with helper methods inside your test suite classes; however, you would need to remember to call both methods in every test case. Minitest provides a better way: each class that defines a test suite can have a #setup and/or a #teardown method to automatically perform any preprocessing or postprocessing required.

The #setup method is called prior to each test case in the class, and is used to perform any setup required for each test. Likewise, #teardown is called after each test case to perform any cleanup required. It is common to set instance variables in #setup that can be used in the actual test case methods. For example:

require 'minitest/autorun'
require 'pg'
class MyApp
  def initialize
    @db = PG.connect 'mydb'
  end
  def cleanup
    @db.finish
  end
  def count; ...; end
  def create(value); ...; end
end
class DatabaseTest < Minitest::Test
  def setup
    @myapp = MyApp.new
  end
  def test_that_query_on_empty_database_returns_nothing
    assert_equal 0, @myapp.count
  end
  def test_that_query_on_non_empty_database_returns_right_count
    @myapp.create('Abc')
    @myapp.create('Def')
    @myapp.create('Ghi')
    assert_equal 3, @myapp.count
  end
  def teardown
    @myapp.cleanup
  end
end

This test suite runs two test cases. Prior to each test case, #setup creates a @myapp instance variable that references a MyApp object. After each test case, #teardown calls @myapp.cleanup to perform any shutdown cleanup required by the MyApp class. In this example, set up and tear down consist of code that establishes and then drops a database connection. Elsewhere in the test suite, we can reference @myapp freely to access the object.

Note that both #setup and #teardown are independent and optional; you can have both, neither, or either one in any test suite.

Testing Error Handling (assert_raises)

Testing involves not only checking that methods produce the correct results when given correct inputs, but should also test that those methods handle error conditions correctly. This is easy when the success or failure of a method can be determined by just testing its return value. However, some methods raise exceptions. To test exceptions, you need the #assert_raises method. We saw an example of this earlier:

def test_with_negative_number
  assert_raises(Math::DomainError) { square_root(-3) }
end

Here, #assert_raises asserts that the associated block ({ square_root(-3) }) should raise a Math::DomainError exception, or an exception that is a subclass of Math::DomainError. If no exception is raised or a different exception is raised, #assert_raises fails and issues an appropriate failure message.

Testing Output

Many tests require that you examine the terminal output of your application. In this section, we’ll examine Minitest’s techniques for testing output.

assert_silent

With many applications, methods, etc, the ideal situation is that no output at all is produced. To test for this situation, simply invoke the method under test in a block that gets passed to #assert_silent:

def test_has_no_output
  assert_silent { update_database }
end

If #update_database prints anything to stdout or stderr, the assertion will fail. If nothing is printed to stdout nor stderr, the assertion passes.

assert_output

Sometimes you need to test what gets printed, and where it gets printed. For example, you may want to ensure that good output is sent to stdout, while error messages are printed to stderr. For this, we use #assert_output:

def test_stdout_and_stderr
  assert_output('', /No records found/) do
    print_all_records
  end
end

The first argument for #assert_output specifies the expected output that will be sent to stdout, while the second argument specifies the expected output for stderr. Each of these arguments can take the following values:

• nil means the assertion doesn't care what gets written to the stream.
• A string means the assertion expects that string to match the output stream exactly. (In particular, an empty string is used to assert that no output is sent to a stream.)
• A regular expression means the assertion should expect a string that matches the regex.

In our example above, the assertion expects that #print_all_records won't print anything to stdout, but will output an error message to stderr that contains the regular expression pattern No records found.

capture_io

An alternative to using #assert_output is to use #capture_io:

def test_stdout_and_stderr
  out, err = capture_io do
    print_all_records
  end
  assert_equal('', out)
  assert_match(/No records found/, err)
end

This is equivalent to the example shown in the previous section. The chief advantage of using capture_io is it lets you handle the stderr and stdout separately. This is especially handy when you want to run multiple tests on one or both of out and err that can't readily be handled with #assert_output.

Testing Classes and Objects

Ruby is pretty lax about types: methods can both accept and return values of different types at different times. Sometimes, it is handy to verify that a value has the expected type, or that it can respond to a particular method. For these cases, Minitest provides several useful methods.

assert_instance_of

#assert_instance_of asserts that an object is an object of a particular class; it is analogous to the standard Ruby method Object#instance_of?.

assert_instance_of SomeClass, object

succeeds if object is a SomeClass object, fails otherwise.

assert_kind_of

#assert_kind_of asserts that an object is an object of a particular class or one of its subclasses; it is analogous to the standard Ruby method Object#kind_of? or Object#is_a?.

assert_kind_of SomeClass, object

succeeds if object is a SomeClass object or one of its subclasses, fails otherwise.

assert_respond_to

In Ruby, you often don’t need to know that an object has a particular type; instead, you’re more interested in what methods an object responds to. Ruby even provides a method, Object#respond_to?, that can be applied to any object to determine if it responds to a given method. This carries over to testing as well: you're often more interested in knowing if an an object responds to a given method, and #assert_respond_to provides this capability:

assert_respond_to object, :empty?

This test asserts that object responds to the method #empty?, e.g., object.empty? is a valid method call. The method name may be specified as a symbol, as shown above, or as a String that will be converted to a symbol internally.

Refutations

Most often, your test cases will be more interested in determining whether or not a specific condition is true. For example, you may want to know whether a method returns true, 5, or 'xyz'. For such situations, assertions are ideal; you write your assertion in such a way that it asserts whether the actual value has the expected value.

Much less often, you will encounter situations in which you are interested in the negation of a condition; for example, if you want to ensure that a method that operates on an Array returns a new Array instead of modifying the original Array, you need to assert that that result Array is not the same object as the original Array. You can write this as:

ary = [...]
assert ary.object_id != method(ary).object_id, 'method(ary) returns original Array'

However, this isn’t particularly clear because we are forced to use the bare assert instead of one of the more specialized assertions like assert_same, which would be easier to read.

For these cases, Minitest provides refutations. Refutations are assertions that assert that something isn’t true. For example, we can write:

ary = [...]
refute(ary.object_id == method(ary).object_id,
       'method(ary) returns copy of original Array')

This simplifies further to:

ary = [...]
refute_equal ary.object_id, method(ary).object_id

Better yet, we can use the negation of assert_same, namely refute_same, to be even more clear:

ary = [...]
refute_same ary, method(ary)

Most of the Minitest assertions have equivalent refutations that test the negation of the condition. In all cases, the refutation uses the assertion name with assert replaced by refute, and arguments for refutations are identical to the arguments for assertions. So, for example, the refutation of:

assert_nil item

is:

refute_nil item

The following assertions do not have a corresponding refutation:

• assert_output
• assert_raises
• assert_send
• assert_silent
• assert_throws

Uncovered Methods

This post isn’t meant to be a complete reference to Minitest; for that, you should refer to the Minitest documentation. However, here’s a short list of some other methods from the Minitest::Assertions module that you may find useful:

• #assert_in_epsilon is similar to #assert_in_delta, except the delta is based on the relative size of the actual or expected value.
• #assert_operator is used to test binary operation such as #<=, #+, etc.
• #assert_predicate is used to test predicates -- usually used by expectations, not assertions.
• #assert_send calls an arbitrary method on an object and asserts that the return value is true.
• #assert_throws tests whether a block returns via a throw to a specific symbol.
• #capture_subprocess_io is similar to #capture_io except that is also captures output of subprocesses. If #capture_io doesn't do what you want, try #capture_subprocess_io instead.
• #flunk forces a test to fail
• #skip forces a test to be skipped
• #skipped? returns true if a test was skipped. Sometimes useful in #teardown.

Testing Startup Code

If the module you are testing has some launch code that starts the program running, you may have to make a small modification to that code. For example, the launch code in an Xyzzy module may look something like this:

Xyzzy.new.run

If you run this code during testing, the results may be confusing as your application will start in addition to being tested. To avoid this, you need to modify the launch code so it doesn’t run when the file is required. You can do this like so:

Xyzzy.new.run if __FILE__ == $PROGRAM_NAME

or

Xyzzy.new.run if __FILE__ == $0

If you run the program directly, both __FILE__ and $PROGRAM_NAME (or $0) reference the program file. If, instead, you require the file into your test module, $PROGRAM_NAME and $0 will be the name of the test program, but __FILE__ will continue to refer to the main program file; since the two names differ, the launch code will not run.

Some programs don’t have any obvious launch code like Xyzzy.new.run; for example, Sinatra applications don't have any obvious launch code. In such cases, you may need to find a different way to prevent running the program when testing. With Sinatra (and Rack::Test), you must run the following before performing any route requests:

ENV['RACK_ENV'] = 'test'

This code is added to your test module.

Conclusion

This concludes our introduction to Minitest, and, in particular, its assertions interface. As we’ve seen, using Minitest is simple and straightforward, with a wide variety of assertions and refutations that can help you write the tests required to determine whether your application works.

This isn’t the complete story about Minitest. In particular, we only briefly mentioned topics such as expectations, mocking and stubs, benchmarking, custom assertions and expectations, and the different reporting and execution options. While we hope to cover some of these topics in future posts, for now you will need to refer to the Minitest documentation.

Wrap-up

We hope you’ve enjoyed this introduction to Minitest, and to its assertions interface in particular. This and many other topics are discussed extensively in the curriculum at Launch School. Logged in users (free registration) can also access many exercises on a wide variety of topics. Feel free to stop by and see what we’re all about.