Chapter 16 - Application Management Tools

During both the development and deployment phases, developers require a consistent stream of diagnostic information in order to determine whether the application is working as intended. This information is generally aggregated through logging and debugging utilities. Because of the central role frameworks, such as symfony, play in driving applications, it's crucial that such capabilities are tightly integrated to ensure efficient developmental and operational activities.

During the life of an application on the production server, the application administrator repeats a large number of tasks, from log rotation to upgrades. A framework must also provide tools to automate these tasks as much as possible.

This chapter explains how symfony application management tools can answer all these needs.

Logging

The only way to understand what went wrong during the execution of a request is to review a trace of the execution process. Fortunately, as you'll learn in this section, both PHP and symfony tend to log large amounts of this sort of data.

PHP Logs

PHP has an error_reporting parameter, defined in php.ini, that specifies which PHP events are logged. Symfony allows you to override this value, per application and environment, in the settings.yml file, as shown in Listing 16-1.

Listing 16-1 - Setting the Error Reporting Level, in myapp/config/settings.yml

prod:
 .settings:
    error_reporting:  257

dev:
  .settings:
    error_reporting:  4095

The numbers are a short way of writing error levels (refer to the PHP documentation on error reporting for more details). Basically, 4095 is a shortcut for E_ALL | E_STRICT, and 257 stands for E_ERROR | E_USER_ERROR (the default value for every new environment).

In order to avoid performance issues in the production environment, the server logs only the critical PHP errors. However, in the development environment, all types of events are logged, so that the developer can have all the information necessary to trace errors.

The location of the PHP log files depends on your php.ini configuration. If you never bothered about defining this location, PHP probably uses the logging facilities provided by your web server (such as the Apache error logs). In this case, you will find the PHP logs under the web server log directory.

Symfony Logs

In addition to the standard PHP logs, symfony can log a lot of custom events. You can find all the symfony logs under the myproject/log/ directory. There is one file per application and per environment. For instance, the development environment log file of the myapp application is named myapp_dev.log, the production one is named myapp_prod.log, and so on.

If you have a symfony application running, take a look at its log files. The syntax is very simple. For every event, one line is added to the log file of the application. Each line includes the exact time of the event, the nature of the event, the object being processed, and any additional relevant details. Listing 16-2 shows an example of symfony log file content.

Listing 16-2 - Sample Symfony Log File Content, in log/myapp_dev.php

Nov 15 16:30:25 symfony [info ] {sfAction} call "barActions->executemessages()"
Nov 15 16:30:25 symfony [debug] SELECT bd_message.ID, bd_message.SENDER_ID, bd_...
Nov 15 16:30:25 symfony [info ] {sfCreole} executeQuery(): SELECT bd_message.ID...
Nov 15 16:30:25 symfony [info ] {sfView} set slot "leftbar" (bar/index)
Nov 15 16:30:25 symfony [info ] {sfView} set slot "messageblock" (bar/mes...
Nov 15 16:30:25 symfony [info ] {sfView} execute view for template "messa...
Nov 15 16:30:25 symfony [info ] {sfView} render "/home/production/myproject/...
Nov 15 16:30:25 symfony [info ] {sfView} render to client

You can find many details in these files, including the actual SQL queries sent to the database, the templates called, the chain of calls between objects, and so on.

Symfony Log Level Configuration

There are eight levels of symfony log messages: emerg, alert, crit, err, warning, notice, info, and debug, which are the same as the PEAR::Log package (http://pear.php.net/package/Log/) levels. You can configure the maximum level to be logged in each environment in the logging.yml configuration file of each application, as demonstrated in Listing 16-3.

Listing 16-3 - Default Logging Configuration, in myapp/config/logging.yml

prod:
  enabled: off
  level:   err
  rotate:  on
  purge:   off

dev:

test:

#all:
#  enabled:  on
#  level:    debug
#  rotate:   off
#  period:   7
#  history:  10
#  purge:    on

By default, in all environments except the production environment, all the messages are logged (up to the least important level, the debug level). In the production environment, logging is disabled by default; if you change enabled to on, only the most important messages (from crit to emerg) appear in the logs.

You can change the logging level in the logging.yml file for each environment to limit the type of logged messages. The rotate, period, history, and purge settings are described in the upcoming "Purging and Rotating Log Files" section.

Adding a Log Message

You can manually add a message in the symfony log file from your code by using one of the techniques described in Listing 16-4.

Listing 16-4 - Adding a Custom Log Message

// From an action
$this->logMessage($message, $level);
 
// From a template
<?php use_helper('Debug') ?>
<?php log_message($message, $level) ?>

$level can have the same values as in the log messages.

Alternatively, to write a message in the log from anywhere in your application, use the sfLogger methods directly, as shown in Listing 16-5. The available methods bear the same names as the log levels.

Listing 16-5 - Adding a Custom Log Message from Anywhere

if (sfConfig::get('sf_logging_enabled'))
{
  sfContext::getInstance()->getLogger()->info($message);
}

require_once('Log.php');
$log = Log::singleton('error_log', PEAR_LOG_TYPE_SYSTEM, 'symfony');
sfLogger::getInstance()->registerLogger($log);

Purging and Rotating Log Files

Don't forget to periodically purge the log/ directory of your applications, because these files have the strange habit of growing by several megabytes in a few days, depending, of course, on your traffic. Symfony provides a special log-purge task for this purpose, which you can launch regularly by hand or put in a cron table. For example, the following command erases the symfony log files in applications and environments where the logging.yml file specifies purge: on (which is the default value):

> symfony log-purge

For both better performance and security, you probably want to store symfony logs in several small files instead of one single large file. The ideal storage strategy for log files is to back up and empty the main log file regularly, but to keep only a limited number of backups. You can enable such a log rotation and specify the parameters in logging.yml. For instance, with a period of 7 days and a history (number of backups) of 10, as shown in Listing 16-6, you would work with one active log file plus ten backup files containing seven days' worth of history each. Whenever the next period of seven days ends, the current active log file goes into backup, and the oldest backup is erased.

Listing 16-6 - Configuring Log Rotation, in myapp/config/logging.yml

prod:
  rotate:  on
  period:  7       ## Log files are rotated every 7 days by default
  history: 10      ## A maximum history of 10 log files is kept

To execute the log rotation, periodically execute the log-rotate task. This task only purges files for which rotate is on. You can specify a single application and environment when calling the task:

> symfony log-rotate myapp prod

The backup log files are stored in the logs/history/ directory and suffixed with the date they were saved.

Debugging

No matter how proficient a coder you are, you will eventually make mistakes, even if you use symfony. Detecting and understanding errors is one of the keys of fast application development. Fortunately, symfony provides several debug tools for the developer.

Symfony Debug Mode

Symfony has a debug mode that facilitates application development and debugging. When it is on, the following happens:

• The configuration is checked at each request, so a change in any of the configuration files has an immediate effect, without any need to clear the configuration cache.
• The error messages display the full stack trace in a clear and useful way, so that you can more efficiently find the faulty element.
• More debug tools are available (such as the detail of database queries).
• The Propel debug mode is also activated, so any error in a call to a Propel object will display a detailed chain of calls through the Propel architecture.

On the other hand, when the debug mode is off, processing is handled as follows:

• The YAML configuration files are parsed only once, then transformed into PHP files stored in the cache/config/ folder. Every request after the first one ignores the YAML files and uses the cached configuration instead. As a consequence, the processing of requests is much faster.
• To allow a reprocessing of the configuration, you must manually clear the configuration cache.
• An error during the processing of the request returns a response with code 500 (Internal Server Error), without any explanation of the internal cause of the problem.

The debug mode is activated per application in the front controller. It is controlled by the value of the SF_DEBUG constant, as shown in Listing 16-7.

Listing 16-7 - Sample Front Controller with Debug Mode On, in web/myapp_dev.php

<?php
 
define('SF_ROOT_DIR',    realpath(dirname(__FILE__).'/..'));
define('SF_APP',         'myapp');
define('SF_ENVIRONMENT', 'dev');
define('SF_DEBUG',       true);
 
require_once(SF_ROOT_DIR.DIRECTORY_SEPARATOR.'apps'.DIRECTORY_SEPARATOR.SF_APP.DIRECTORY_SEPARATOR.'config'.DIRECTORY_SEPARATOR.'config.php');
 
sfContext::getInstance()->getController()->dispatch();

Symfony Exceptions

When an exception occurs in the debug mode, symfony displays a useful exception notice that contains everything you need to find the cause of the problem.

The exception messages are clearly written and refer to the most probable cause of the problem. They often provide possible solutions to fix the problem, and for most common problems, the exception pages even contain a link to a symfony website page with more details about the exception. The exception page shows where the error occurred in the PHP code (with syntax highlighting), together with the full stack of method calls, as shown in Figure 16-1. You can follow the trace to the first call that caused the problem. The arguments that were passed to the methods are also shown.

Figure 16-1 - Sample exception message for a symfony application

During the development phase, the symfony exceptions will be of great use as you debug your application.

Xdebug Extension

The Xdebug PHP extension (http://xdebug.org/) allows you to extend the amount of information that is logged by the web server. Symfony integrates the Xdebug messages in its own debug feedback, so it is a good idea to activate this extension when you debug the application. The extension installation depends very much on your platform; refer to the Xdebug website for detailed installation guidelines. Once Xdebug is installed, you need to activate it manually in your php.ini file after installation. For *nix platforms, this is done by adding the following line:

zend_extension="/usr/local/lib/php/extensions/no-debug-non-zts-20041030/xdebug.so"

For Windows platforms, the Xdebug activation is triggered by this line:

extension=php_xdebug.dll

Listing 16-8 gives an example of Xdebug configuration, which must also be added to the php.ini file.

Listing 16-8 - Sample Xdebug Configuration

;xdebug.profiler_enable=1
;xdebug.profiler_output_dir="/tmp/xdebug"
xdebug.auto_trace=1             ; enable tracing
xdebug.trace_format=0
;xdebug.show_mem_delta=0        ; memory difference
;xdebug.show_local_vars=1
;xdebug.max_nesting_level=100

You must restart your web server for the Xdebug mode to be activated.

Web Debug Toolbar

The log files contain interesting information, but they are not very easy to read. The most basic task, which is to find the lines logged for a particular request, can be quite tricky if you have several users simultaneously using an application and a long history of events. That's when you start to need a web debug toolbar.

This toolbar appears as a semitransparent box superimposed over the normal content in the browser, in the top-right corner of the window, as shown in Figure 16-2. It gives access to the symfony log events, the current configuration, the properties of the request and response objects, the details of the database queries issued by the request, and a chart of processing times related to the request.

Figure 16-2 - The web debug toolbar appears in the top-right corner of the window

The color of the debug toolbar background depends on the highest level of log message issued during the request. If no message passes the debug level, the toolbar has a gray background. If a single message reaches the err level, the toolbar has a red background.

To activate the web debug toolbar for an application, open the settings.yml file and look for the web_debug key. In the prod and test environments, the default value for web_debug is off, so you need to activate it manually if you want it. In the dev environment, the default configuration has it set to on, as shown in Listing 16-9.

Listing 16-9 - Web Debug Toolbar Activation, in myapp/config/settings.yml

dev:
  .settings:
    web_debug:              on

When displayed, the web debug toolbar offers a lot of information/interaction:

• Click the symfony logo to toggle the visibility of the toolbar. When reduced, the toolbar doesn't hide the elements located at the top of the page.
• Click the vars & config section to show the details of the request, response, settings, globals, and PHP properties, as shown in Figure 16-3. The top line sums up the important configuration settings, such as the debug mode, the cache, and the presence of a PHP accelerator (they appear in red if they are deactivated and in green if they are activated).

Figure 16-3 - The vars & config section shows all the variables and constants of the request

• When the cache is enabled, a green arrow appears in the toolbar. Click this arrow to reprocess the page, regardless of what is stored in the cache (but the cache is not cleared).
• Click the logs & msgs section to reveal the log messages for the current request, as shown in Figure 16-4. According to the importance of the events, they are displayed in gray, yellow, or red lines. You can filter the events that are displayed by category using the links displayed at the top of the list.

Figure 16-4 - The logs & msgs section shows the log messages for the current request

• For requests executing SQL queries, a database icon appears in the toolbar. Click it to see the detail of the queries, as shown in Figure 16-5.
• To the right of a clock icon is the total time necessary to process the request. Be aware that the web debug toolbar and the debug mode slow down the request execution, so try to refrain from considering the timings per se, and pay attention to only the differences between the execution time of two pages. Click the clock icon to see details of the processing time category by category, as shown in Figure 16-6. Symfony displays the time spent on specific parts of the request processing. Only the times related to the current request make sense for an optimization, so the time spent in the symfony core is not displayed. That's why these times don't sum up to the total time.
• Click the red x at the right end of the toolbar to hide the toolbar.

Figure 16-5 - The database queries section shows queries executed for the current request

Figure 16-6 - The clock icon shows execution time by category

// Initialize the timer and start timing
$timer = sfTimerManager::getTimer('myTimer');
 
// Do things
...
 
// Stop the timer and add the elapsed time
$timer->addTime();
 
// Get the result (and stop the timer if not already stopped)
$elapsedTime = $timer->getElapsedTime();

// Get the number of calls to the timer
$nbCalls = $timer->getCalls();

Manual Debugging

Getting access to the framework debug messages is nice, but being able to log your own messages is better. Symfony provides shortcuts, accessible from both actions and templates, to help you trace events and/or values during request execution.

Your custom log messages appear in the symfony log file as well as in the web debug toolbar, just like regular events. (Listing 16-4 gave an example of the custom log message syntax.) A custom message is a good way to check the value of a variable from a template, for instance. Listing 16-10 shows how to use the web debug toolbar for developer's feedback from a template (you can also use $this->logMessage() from an action).

Listing 16-10 - Inserting a Message in the Log for Debugging Purposes

<?php use_helper('Debug') ?>
...
<?php if ($problem): ?>
  <?php log_message('{sfAction} been there', 'err') ?>
  ...
<?php endif ?>

The use of the err level guarantees that the event will be clearly visible in the list of messages, as shown in Figure 16-7.

Figure 16-7 - A custom log message appears in the logs & msgs section of the web debug toolbar

If you don't want to add a line to the log, but just trace a short message or a value, you should use debug_message instead of log_message. This action method (a helper with the same name also exists) displays a message in the web debug toolbar, on top of the logs & msgs section. Check Listing 16-11 for an example of using the debug message writer.

Listing 16-11 - Inserting a Message in the Debug Toolbar

// From an action
$this->debugMessage($message);
 
// From a template
<?php use_helper('Debug') ?>
<?php debug_message($message) ?>

Populating a Database

In the process of application development, developers are often faced with the problem of database population. A few specific solutions exist for some database systems, but none can be used on top of the object-relational mapping. Thanks to YAML and the sfPropelData object, symfony can automatically transfer data from a text source to a database. Although writing a text file source for data may seem like more work than entering the records by hand using a CRUD interface, it will save you time in the long run. You will find this feature very useful for automatically storing and populating the test data for your application.

Fixture File Syntax

Symfony can read data files that follow a very simple YAML syntax, provided that they are located under the data/fixtures/ directory. Fixture files are organized by class, each class section being introduced by the class name as a header. For each class, records labeled with a unique string are defined by a set of fieldname: value pairs. Listing 16-12 shows an example of a data file for database population.

Listing 16-12 - Sample Fixture File, in data/fixtures/import_data.yml

Article:                             ## Insert records in the blog_article table
  first_post:                        ## First record label
    title:       My first memories
    content: |
      For a long time I used to go to bed early. Sometimes, when I had put
      out my candle, my eyes would close so quickly that I had not even time
      to say "I'm going to sleep".

  second_post:                       ## Second record label
    title:       Things got worse
    content: |
      Sometimes he hoped that she would die, painlessly, in some accident,
      she who was out of doors in the streets, crossing busy thoroughfares,
      from morning to night.

Symfony translates the column keys into setter methods by using a camelCase converter (setTitle(), setContent()). This means that you can define a password key even if the actual table doesn't have a password field--just define a setPassword() method in the User object, and you can populate other columns based on the password (for instance, a hashed version of the password).

The primary key column doesn't need to be defined. Since it is an auto-increment field, the database layer knows how to determine it.

The created_at columns don't need to be set either, because symfony knows that fields named that way must be set to the current system time when created.

Launching the Import

The propel-load-data task imports data from a YAML file to a database. The connection settings come from the databases.yml file, and therefore need an application name to run. Optionally, you can specify an environment name (dev by default).

> symfony propel-load-data frontend

This command reads all the YAML fixture files from the data/fixtures/ directory and inserts the records into the database. By default, it replaces the existing database content, but if the last argument call is append, the command will not erase the current data.

> symfony propel-load-data frontend append

You can specify another fixture file or directory in the call. In this case, add a path relative to the project directory.

> symfony propel-load-data frontend data/myfixtures/myfile.yml

Using Linked Tables

You now know how to add records to a single table, but how do you add records with foreign keys to another table? Since the primary key is not included in the fixtures data, you need an alternative way to relate records to one another.

Let's return to the example in Chapter 8, where a blog_article table is linked to a blog_comment table, as shown in Figure 16-8.

Figure 16-8 - A sample database relational model

This is where the labels given to the records become really useful. To add a Comment field to the first_post article, you simply need to append the lines shown in Listing 16-13 to the import_data.yml data file.

Listing 16-13 - Adding a Record to a Related Table, in data/fixtures/import_data.yml

Comment:
  first_comment:
    article_id:   first_post
    author:       Anonymous
    content:      Your prose is too verbose. Write shorter sentences.

The propel-load-data task will recognize the label that you gave to an article previously in import_data.yml, and grab the primary key of the corresponding Article record to set the article_id field. You don't even see the IDs of the records; you just link them by their labels--it couldn't be simpler.

The only constraint for linked records is that the objects called in a foreign key must be defined earlier in the file; that is, as you would do if you defined them one by one. The data files are parsed from the top to the bottom, and the order in which the records are written is important.

One data file can contain declarations of several classes. But if you need to insert a lot of data for many different tables, your fixture file might get too long to be easily manipulated.

The propel-load-data task parses all the files it finds in the fixtures/ directory, so nothing prevents you from splitting a YAML fixture file into smaller pieces. The important thing to keep in mind is that foreign keys impose a processing order for the tables. To make sure that they are parsed in the correct order, prefix the files with an ordinal number.

100_article_import_data.yml
200_comment_import_data.yml
300_rating_import_data.yml

Deploying Applications

Symfony offers shorthand commands to synchronize two versions of a website. These commands are mostly used to deploy a website from a development server to a final host, connected to the Internet.

Freezing a Project for FTP Transfer

The most common way to deploy a project to production is to transfer all its files by FTP (or SFTP). However, symfony projects use the symfony libraries, and unless you develop in a sandbox (which is not recommended), or if the symfony lib/ and data/ directories are linked by svn:externals, these libraries are not in the project directory. Whether you use a PEAR installation or symbolic links, reproducing the same file structure in production can be time-consuming and tricky.

That's why symfony provides a utility to "freeze" a project--to copy all the necessary symfony libraries into the project data/, lib/, and web/ directories. The project then becomes a kind of sandbox, an independent, stand-alone application.

> symfony freeze

Once a project is frozen, you can transfer the project directory into production, and it will work without any need for PEAR, symbolic links, or whatever else.

To revert a project to its initial state, use the unfreeze task. It erases the data/symfony/, lib/symfony/, and web/sf/ directories.

> symfony unfreeze

Note that if you had symbolic links to a symfony installation prior to the freeze, symfony will remember them and re-create the symbolic links in the original location.

Using rsync for Incremental File Transfer

Sending the root project directory by FTP is fine for the first transfer, but when you need to upload an update of your application, where only a few files have changed, FTP is not ideal. You need to either transfer the whole project again, which is a waste of time and bandwidth, or browse to the directories where you know that some files changed, and transfer only the ones with different modification dates. That's a time-consuming job, and it is prone to error. In addition, the website can be unavailable or buggy during the time of the transfer.

The solution that is supported by symfony is rsync synchronization through an SSH layer. Rsync (http://samba.anu.edu.au/rsync/) is a command-line utility that provides fast incremental file transfer, and it's open source. With incremental transfer, only the modified data will be sent. If a file didn't change, it won't be sent to the host. If a file changed only partially, just the differential will be sent. The major advantage is that rsync synchronizations transfer only a small amount of data and are very fast.

Symfony adds SSH on top of rsync to secure the data transfer. More and more commercial hosts support an SSH tunnel to secure file uploads on their servers, and that's a good practice to avoid security breaches.

The SSH client called by symfony uses connection settings from the config/properties.ini file. Listing 16-14 gives an example of connection settings for a production server. Write the settings of your own production server in this file before any synchronization. You can also define a single parameters setting to provide your own rsync command line parameters.

Listing 16-14 - Sample Connection Settings for a Server Synchronization, in myproject/config/properties.ini

name=myproject
 
[production]
  host=myapp.example.com
  port=22
  user=myuser
  dir=/home/myaccount/myproject/

Doing an rsync over SSH requires several commands, and synchronization can occur a lot of times in the life of an application. Fortunately, symfony automates this process with just one command:

> symfony sync production

This command launches the rsync command in dry mode; that is, it shows which files must be synchronized but doesn't actually synchronize them. If you want the synchronization to be done, you need to request it explicitly by adding go.

> symfony sync production go

Don't forget to clear the cache in the production server after synchronization.

Ignoring Irrelevant Files

If you synchronize your symfony project with a production host, a few files and directories should not be transferred:

• All the version control directories (.svn/, CVS/, and so on) and their content are necessary only for development and integration.
• The front controller for the development environment must not be available to the final users. The debugging and logging tools available when using the application through this front controller slow down the application and give information about the core variables of your actions. It is something to keep away from the public.
• The cache/ and log/ directories of a project must not be erased in the host server each time you do a synchronization. These directories must be ignored as well. If you have a stats/ directory, it should probably be treated the same way.
• The files uploaded by users should not be transferred. One of the good practices of symfony projects is to store the uploaded files in the web/uploads/ directory. This allows you to exclude all these files from the synchronization by pointing to only one directory.

To exclude files from rsync synchronizations, open and edit the rsync_exclude.txt file under the myproject/config/ directory. Each line can contain a file, a directory, or a pattern. The symfony file structure is organized logically, and designed to minimize the number of files or directories to exclude manually from the synchronization. See Listing 16-15 for an example.

Listing 16-15 - Sample rsync Exclusion Settings, in myproject/config/rsync_exclude.txt

.svn
/cache/*
/log/*
/stats/*
/web/uploads/*
/web/myapp_dev.php

Managing a Production Application

The command that is used most often in production servers is clear-cache. You must run it every time you upgrade symfony or your project (for instance, after calling the sync task), and every time you change the configuration in production.

> symfony clear-cache

You can temporarily disable your application--for instance, when you need to upgrade a library or a large amount of data.

> symfony disable APPLICATION_NAME ENVIRONMENT_NAME

By default, a disabled application displays the default/unavailable action (stored in the framework), but you can customize the module and action to be used in this case in the settings.yml file. Listing 16-16 shows an example.

Listing 16-16 - Setting the Action to Execute for an Unavailable Application, in myapp/config/settings.yml

all:
  .settings:
    unavailable_module:     mymodule
    unavailable_action:     maintenance

The enable task reenables the application and clears the cache.

> symfony enable APPLICATION_NAME ENVIRONMENT_NAME

> symfony clear-controllers

> symfony fix-perms

// With symfony installed via PEAR
> symfony [options] <TASK> [parameters]

// With symfony frozen in the project or symlinked
> php symfony [options] <TASK> [parameters]

Summary

By combining PHP logs and symfony logs, you can monitor and debug your application easily. During development, the debug mode, the exceptions, and the web debug toolbar help you locate problems. You can even insert custom messages in the log files or in the toolbar for easier debugging.

The command-line interface provides a large number of tools that facilitate the management of your applications, during development and production phases. Among others, the data population, freeze, and synchronization tasks are great time-savers.