Day 15: Unit tests

Previously on symfony

The questions are now well organized in the askeet website, thanks to the community tagging feature that we added yesterday.

But there is a thing that has not been described until now, despite its importance in the life of web applications. Unit tests are one of the greatest advances in programming since object orientation. They allow for a safe development process, refactoring without fear, and can sometimes replace documentation since they illustrate quite clearly what an application is supposed to do. Symfony supports and recommends unit testing, and provides tools for that. The overview of these tools - and the addition of a few unit tests to askeet - will take much of our time today.

Simple test

There are many unit test frameworks in the PHP world, mostly based on Junit. We didn't develop another one for symfony, but instead we integrated the most mature of them all, Simple Test. It is stable, well documented, and offers tons of features that are of considerable value for all PHP projects, including symfony ones. If you don't know it already, you are strongly advised to browse their documentation, which is very clear and progressive.

Simple Test is not bundled with symfony, but very simple to install. First, download the Simple Test PEAR installable archive at SourceForge. Install it via pear by calling:

$ pear install simpletest_1.0.0.tgz

If you want to write a batch script that uses the Simple Test library, all you have to do is insert these few lines of code on top of the script:

<?php
 
require_once('simpletest/unit_tester.php');
require_once('simpletest/reporter.php');
 
?>

Symfony does it for you if you use the test command line; we will talk about it shortly.

Unit tests in a symfony project

Default unit tests

Each symfony project has a test/ directory, divided into application subdirectories. For askeet, if you browse to the askeet/test/functional/frontend/ directory, you will see that a few files already exist there:

answerActionsTest.php
feedActionsTest.php
mailActionsTest.php
sidebarActionsTest.php
userActionsTest.php

They all contain the same initial code:

<?php
 
class answerActionsWebBrowserTest extends UnitTestCase
{
  private
    $browser = null;
 
  public function setUp ()
  {
    // create a new test browser
    $this->browser = new sfTestBrowser();
    $this->browser->initialize('hostname');
  }
 
  public function tearDown ()
  {
    $this->browser->shutdown();
  }
 
  public function test_simple()
  {
    $url = '/answer/index';
    $html = $this->browser->get($url);
    $this->assertWantedPattern('/answer/', $html);
  }
}
 
?>

The UnitTestCase class is the core class of the Simple Test unit tests. The setUp() method is run just before each test method, and tearDown() is run just after each test method. The actual test methods start with the word 'test'. To check if a piece of code is behaving as you expect, you use an assertion, which is a method call that verifies that something is true. In Simple Test, assertions start by assert. In this example, one unit test is implemented, and it looks for the word 'user' in the default page of the module. This autogenerated file is a stub for you to start.

As a matter of fact, every time you call a symfony init-module, symfony creates a skeleton like this one in the test/[appname]/ directory to store the unit tests related to the created module. The trouble is that as soon as you modify the default template, the stub tests don't pass anymore (they check the default title of the page, which is 'module $modulename'). So for now, we will erase these files and work on our own test cases.

Add a unit test

During day 13, we created a Tag.class.php file with two functions dedicated to tag manipulation. We will add a few unit tests for our Tag library.

Create a TagTest.php file (all the test case files must end with Test for Simple Test to find them):

<?php
 
require_once('Tag.class.php');
 
class TagTest extends UnitTestCase
{
  public function test_normalize()
  {
    $tests = array(
      'FOO'       => 'foo',
      '   foo'    => 'foo',
      'foo  '     => 'foo',
      ' foo '     => 'foo',
      'foo-bar'   => 'foobar',
    );
 
    foreach ($tests as $tag => $normalized_tag)
    {
      $this->assertEqual($normalized_tag, Tag::normalize($tag));
    }
  }
}
 
?>

The first test case that we will implement concerns the Tag::normalize() method. Unit tests are supposed to test one case at a time, so we decompose the expected result of the text method into elementary cases. We know that the Tag::normalize() method is supposed to return a lower-case version of its argument, without any spaces - either before or after the argument - and without any special character. The five test cases defined in the $test array are enough to test that.

For each of the elementary test cases, we then compare the normalized version of the input with the expected result, with a call to the ->assertEqual() method. This is the heart of a unit test. If it fails, the name of the test case will be output when the test suite is run. If it passes, it will simply add to the number of passed tests.

We could add a last test with the word ' FOo-bar ', but it mixes elementary cases. If this test fails, you won't have a clear idea of the precise cause of the problem, and you will need to investigate further. Keeping to elementary cases gives you the insurance that the error will be located easily.

Running unit tests

The symfony command line allows you to run all the tests at once with a single command (remember to call it from your project root directory):

$ symfony test-functional frontend

Calling this command executes all the tests of the test/functional/frontend/ directory, and for now it is only the ones of our new TagTest.php set. These tests will pass and the command line will show:

$ symfony test-functional frontend
Test suite in (test/frontend)
OK
Test cases run: 1/1, Passes: 5, Failures: 0, Exceptions: 0

The other way around

The greatest benefit of unit tests is experienced when doing test-driven development. In this methodology, the tests are written before the function is written.

With the example above, you would write an empty Tag::normalize() method, then write the first test case ('Foo'/'foo'), then run the test suite. The test would fail. You would then add the necessary code to transform the argument into lowercase and return it in the Tag::normalize() method, then run the test again. The test would pass this time.

So you would add the tests for blanks, run them, see that they fail, add the code to remove the blanks, run the tests again, see that they pass. Then do the same for the special characters.

Writing tests first helps you to focus on the things that a function should do before actually developing it. It's a good practice that others methodologies, like eXtreme Programming, recommend as well. Plus it takes into account the undeniable fact that if you don't write unit tests first, you never write them.

One last recommendation: keep your unit tests as simple as the ones described here. An application built with a test driven methodology ends up with roughly as much test code as actual code, so you don't want to spend time debugging your tests cases...

When a test fails

We will now add the tests to check the second method of the Tag object, which splits a string made of several tags into an array of tags. Add the following method to the TagTest class:

public function test_splitPhrase()
{
  $tests = array(
    'foo'              => array('foo'),
    'foo bar'          => array('foo', 'bar'),
    '  foo    bar  '   => array('foo', 'bar'),
    '"foo bar" askeet' => array('foo bar', 'askeet'),
    "'foo bar' askeet" => array('foo bar', 'askeet'),
  );
 
  foreach ($tests as $tag => $tags)
  {
    $this->assertEqual($tags, Tag::splitPhrase($tag));
  }
}

If you try to run the tests again, they fail:

$ symfony test-functional frontend
Test suite in (test/frontend)
1) Equal expectation fails as key list [0, 1] does not match key list [0, 1, 2] at line [35]
        in test_splitPhrase
        in TagTest
        in /home/production/askeet/test/functional/frontend/TagTest.php
FAILURES!!!
Test cases run: 1/1, Passes: 9, Failures: 1, Exceptions: 0

All right, one of the test cases of test_splitPhrase fails. To find which one it is, you will need to remove them one at at time to see when the test passes. This time, it's the last one, when we test the handling of simple quotes. The current Tag::splitPhrase() method doesn't translate this string properly. As part of your homework, you will have to correct it for tomorrow.

This illustrates the fact that if you pile up too much elementary test cases in an array, a failure is harder to locate. Always prefer to split long test cases into methods, since Simple Test mentions the name of the method where a test failed.

Simulating a web browsing session

Web applications are not all about objects that behave more or less like functions. The complex mechanisms of page request, HTML result and browser interactions require more than what's been exposed before to build a complete set of unit tests for a symfony web app.

We will examine three different ways to implement a simple web app test. The test has to do a request to the first question detail, and assume that some text of the answer is present. We will put this test into a QuestionTest.php file, located in the askeet/test/functional/frontend/ directory.

The sfTestBrowser object

Symfony provides an object called sfTestBrowser, which allows to simulate browsing without a browser and, more important, without a web server. Being inside the framework allows this object to bypass completely the http transport layer. This means that the browsing simulated by the sfTestBrowser is fast, and independent of the server configuration, since it does not use it.

Let's see how to do a request for a page with this object:

$browser = new sfTestBrowser();
$browser->initialize();
$html = $browser->get('uri');
 
// do some test on $html
 
$browser->shutdown();

The get() request takes a routed URI as a parameter (not an internal URI), and returns a raw HTML page (a string). You can then proceed to all kinds of tests on this page, using the assert*() methods of the UnitTestCase object.

You can pass parameters to your call as you would in the URL bar of your browser:

$html = $browser->get('/frontend_test.php/question/what-can-i-offer-to-my-stepmother');

The reason why we use a specific front controller (frontend_test.php) will be explained in the next section.

The sfTestBrowser simulates a cookie. This means that with a single sfTestBrowser object, you can require several pages one after the other, and they will be considered as part of a single session by the framework. In addition, the fact that sfTestBrowser uses routed URIs instead of internal URIs allows you to test the routing engine.

To implement our web test, the test_QuestionShow() method must be built as follows:

<?php
 
class QuestionTest extends UnitTestCase
{
  public function test_QuestionShow()
  {
    $browser = new sfTestBrowser();
    $browser->initialize();
    $html = $browser->get('frontend_test.php/question/what-can-i-offer-to-my-step-mother');
    $this->assertWantedPattern('/My stepmother has everything a stepmother is usually offered/', $html);
    $browser->shutdown();
  }
}

Since almost all the web unit tests will need a new sfTestBrowser to be initialized and closed after the test, you'd better move part of the code to the ->setUp() and ->tearDown() methods:

<?php 
 
class QuestionTest extends UnitTestCase
{
  private $browser = null;
 
  public function setUp()
  {
    $this->browser = new sfTestBrowser();
    $this->browser->initialize();
  }
 
  public function tearDown()
  {
    $this->browser->shutdown();
  }
 
  public function test_QuestionShow()
  {
    $html = $this->browser->get('frontend_test.php/question/what-can-i-offer-to-my-step-mother');
    $this->assertWantedPattern('/My stepmother has everything a stepmother is usually offered/', $html);
  }
}

Now, every new test method that you add will have a clean sfTestBrowser object to start with. You may recognize here the auto-generated test cases mentioned at the beginning of this tutorial.

The WebTestCase object

Simple Test ships with a WebTestCase class, which includes facilities for navigation, content and cookie checks, and form handling. Tests extending this class allow you to simulate a browsing session with a http transport layer. Once again, the Simple Test documentation explains in detail how to use this class.

The tests built with WebTestCase are slower than the ones built with sfTestBrowser, since the web server is in the middle of every request. They also require that you have a working web server configuration. However, the WebTestCase object comes with numerous navigation methods on top of the assert*() ones. Using these methods, you can simulate a complex browsing session. Here is a subset of the WebTestCase navigation methods:

We could easily do the same test case as previously with a WebTestCase. Beware that you now need to enter full URIs, since they will be requested to the web server:

require_once('simpletest/web_tester.php');
 
class QuestionTest extends WebTestCase
{ 
  public function test_QuestionShow()
  {
    $this->get('http://askeet/frontend_test.php/question/what-can-i-offer-to-my-step-mother');
    $this->assertWantedPattern('/My stepmother has everything a stepmother is usually offered/');
  }
}

The additional methods of this object could help us test how a submitted form is handled, for instance to unit test the login process:

public function test_QuestionAdd()
{
  $this->get('http://askeet/frontend_dev.php/');
  $this->assertLink('sign in/register');
  $this->clickLink('sign in/register');
  $this->assertWantedPattern('/nickname:/');
  $this->setField('nickname', 'fabpot');
  $this->setField('password', 'symfony');
  $this->clickSubmit('sign in');
  $this->assertWantedPattern('/fabpot profile/');      
}

It is very handy to be able to set a value for fields and submit the form as you would do by hand. If you had to simulate that by doing a POST request (and this is possible by a call to ->post($uri, $parameters)), you would have to write in the test function the target of the action and all the hidden fields, thus depending too much on the implementation. For more information about form test with Simple Test, read the related chapter of the Simple Test documentation.

Selenium

The main drawback of both the sfTestBrowser and the WebTestCase tests is that they cannot simulate JavaScript. For very complex interactions, like with AJAX interactions for instance, you need to be able to reproduce exactly the mouse and keyboard inputs that a user would do. Usually, these tests are reproduced by hand, but they are very time consuming and prone to error.

The solution, this time, comes from the JavaScript world. It is called Selenium and is better when employed with the Selenium Recorder extension for Firefox. Selenium executes a set of action on a page just like a regular user would, using the current browser window.

Selenium is not bundled with symfony by default. To install it, you need to create a new selenium/ directory in your web/ directory, and unpack there the content of the Selenium archive. This is because Selenium relies on JavaScript, and the security settings standard in most browsers wouldn't allow it to run unless it is available on the same host and port as your application.

Selenium tests are written in HTML and stored in the selenium/tests/ directory. For instance, to do the simple unit test about question detail, create the following file called testQuestion.html:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html; charset=UTF-8" http-equiv="content-type">
  <title>Question tests</title>
</head>
<body>
<table cellspacing="0">
<tbody>
  <tr><td colspan="3">First step</td></tr>
 
  <tr>
    <td>open</td>
    <td>/frontend_test.php/</td>
    <td>&nbsp;</td>
  </tr>
 
  <tr>
    <td>clickAndWait</td>
    <td>link=What can I offer to my step mother?</td>
    <td>&nbsp;</td>
  </tr>
 
  <tr>
    <td>assertTextPresent</td>
    <td>My stepmother has everything a stepmother is usually offered</td>
    <td>&nbsp;</td>
  </tr>
 
</tbody>
</table>
</body>
</html>

A test-case is represented by an HTML document, containing a table with 3 columns: command, target, value. Not all commands take a value, however. In this case either leave the column blank or use a   to make the table look better.

You also need to add this test to the global test suite by inserting a new line in the table of the TestSuite.html file, located in the same directory:

...
<tr><td><a href='./testQuestion.html'>My First Test</a></td></tr>
...

To run the test, simply browse to

http://askeet/selenium/index.html

Select 'Main Test Suite', than click on the button to run all tests, and watch your browser as it reproduces the steps that you have told him to do.

The fact that Selenium tests are written in HTML could make the writing of Selenium tests a hassle. But thanks to the Firefox Selenium extension, all it takes to create a test is to execute the test once in a recorded session. While navigating in a recording session, you can add assert-type tests by right clicking in the browser window and selecting the appropriate check under the Append Selenium Command in the pop-up menu.

For instance, the following Selenium test checks the AJAX rating of a question. The user 'fabpot' logs in, displays the second page of questions to access the only one he's not interested in so far, then clicks the 'interested?' link, and checks that it changes the '?' into a '!'. It was all recorded with the Firefox extension, and it took less than 30 seconds:

<html>
<head><title>New Test</title></head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
<thead>
<tr><td rowspan="1" colspan="3">New Test</td></tr>
</thead><tbody>
<tr>
    <td>open</td>
    <td>/frontend_dev.php/</td>
    <td></td>
</tr>
<tr>
    <td>clickAndWait</td>
    <td>link=sign in/register</td>
    <td></td>
</tr>
<tr>
    <td>type</td>
    <td>//div/input[@value="" and @id="nickname" and @name="nickname"]</td>
    <td>fabpot</td>
</tr>
<tr>
    <td>type</td>
    <td>//div/input[@value="" and @id="password" and @name="password"]</td>
    <td>symfony</td>
</tr>
<tr>
    <td>clickAndWait</td>
    <td>//input[@type='submit' and @value='sign in']</td>
    <td></td>
</tr>
<tr>
    <td>clickAndWait</td>
    <td>link=2</td>
    <td></td>
</tr>
<tr>
    <td>click</td>
    <td>link=interested?</td>
    <td></td>
</tr>
<tr>
    <td>pause</td>
    <td>3000</td>
    <td></td>
</tr>
<tr>
    <td>verifyTextPresent</td>
    <td>interested!</td>
    <td></td>
</tr>
<tr>
    <td>clickAndWait</td>
    <td>link=sign out</td>
    <td></td>
</tr>
 
</tbody></table>
</body>
</html>

Don't forget to reinitialize the test data (by calling php batch/load_data.php) before launching the Selenium test.

You can save the test to a HTML file to build a Test Suite for your application. The Firefox extension even allows you to run the Selenium tests that you have recorded with it.

A few words about environments

Web tests have to use a front controller, and as such can use a specific environment (i.e. configuration). Symfony provides a test environment to every application by default, specifically for unit tests. You can define a custom set of settings for it in your application config/ directory. The default configuration parameters are (extract from askeet/apps/frontend/config/settings.yml):

test:
  .settings:
    # E_ALL | E_STRICT & ~E_NOTICE = 2047
    error_reporting:        2047
    cache:                  off
    stats:                  off
    web_debug:              off

The cache, the stats and the web_debug toolbar are set to off. However, the code execution still leaves traces in a log file (askeet/log/frontend_test.log). You can have specific database connection settings, for instance to use another database with test data in it.

This is why all the external URIs mentioned above show a frontend_test.php: the test front controller has to be specified - otherwise, the default index.php production controller will be used in place, and you won't be able to use a different database or to have separate logs for your unit tests.

See you Tomorrow

There is no perfect solution for unit testing PHP applications built with symfony for now. Each of the three solutions presented today have great advantages, but if you have an extensive approach of unit testing, you will probably need to use all the three. As for askeet, unit tests will be added little by little in the SVN source. Check for it every now and then, or propose your own to increase the solidity of the application.

Unit testing can also be used to avoid regression. Refactoring a method can create new bugs that didn't use to appear before. That's why it is also a good practice to run all unit tests before deploying a new realease of an application in production - this is called regression testing. We will talk more about it when we deal with application deployement.

Tomorrow... well, tomorrow will be another day. If you have any questions about today's tutorial, feel free to ask them in the askeet forum.