Day 22: Transfer to production

Previously on symfony

Yesterday, we added a back-office to askeet. So everything is ready for the application to actually run and be released on the Internet (by the way, it is already online, try browsing to www.askeet.com if you didn't do it already). This is the perfect moment to focus on the techniques involved in the synchronization of two servers, since you developed askeet on your computer and will probably host it in another server, connected to the Internet.

Synchronization

Good practices

There are a lot of ways to synchronize two environments for a website. The basic file transfers can be achieved by an FTP connection, but there are two major drawbacks to this solution. First, it is not secure, the data stream transmits in the clear over the Internet and can be intercepted. Second, sending the root project directory by FTP is fine for the first transfer, but when you have to upload an update of your application, where only a few files changed, this is not a good and fast way to do it. Either you transfer the whole project again which, can be long or you browse to the directories where you know that some files changed, and transfer only the ones with different modification dates. That's a long job, and it is prone to error. In addition, the website can be unavailable or buggy during the time of the transfer.

The solution that is supported by symfony is rsync synchronization through a SSH layer.

Rsync is a command line utility that provides fast incremental file transfer, and it's open source. By 'incremental', it means that only the modified data will be transferred. If a file didn't change, it won't be sent to the host. If a file changed only partially, only the differential will be sent. The major advantages is that rsync synchronizations transfer only a little data and are very fast.

Symfony adds SSH on top of rsync to secure the data transfer. More and more commercial hosts support an SSH tunnel to secure file uploads on their servers, and that's a good practice that symfony encourages.

For notes on installing rsync and SSH on Linux, read the instructions in the related websites. For Windows users, an open-source alternative called cwRsync exists, or you can try to install the binaries by hand (instructions can be found here). Of course, to be able to setup an SSH tunnel between an integration server and a host server, the SSH service has to be installed and running on both computers.

The symfony sync command

Doing a rsync over SSH requires several commands, and synchronization can occur a lot of times in the life of an application. Fortunately, symfony automates this process with just one command:

$ symfony sync production

This command, called from the root directory of a symfony project, launches the synchronization of the project code with the production hosted server. The connection details of this server are to be written in the project properties.ini, found in askeet/config/:

name=askeet
 
[production]
  host=myaskeetprodserver.com
  port=22
  user=myuser
  dir=/home/myaccount/askeet/

The connection settings will be used by the SSH client call enclosed in the symfony sync command line.

If you just call symfony sync like mentioned above, the rsync utility will run in dry mode by default (--dry-run), i. e. it will show you which files have to be synchronized but without actually synchronizing them. If you want the synchronization to be done, you have to mention it explicitly:

$ symfony sync production go

Ignoring irrelevant files

If you synchronize your symfony project with a production host, there are a few files and directories that should not be transferred:

• All the .svn directories and their content: They contain source version control information, only necessary for development and integration
• askeet/web/frontend_dev.php: The web front controller for the development environment must not be available to the final users. The debugging and logging tools available when using the application through this front controller slow down the application, and give information about the core variables of your actions. It is something to keep off of the host server.
• The cache/ and log/ directories of a project must not be erased in the host server each time you do a synchronization. These directories must be ignored as well. If you have a stats/ directory, it should probably be treated the same.
• The files uploaded by users: one of the good practices of symfony projects is to store the uploaded files in the web/uploads/ directory. This allows us to exclude all these files by pointing to only one directory.

To exclude files from rsync synchronizations, open and edit the rsync_exclude.txt file under the askeet/config/ directory. Each line can contain either a file, a directory, or a pattern:

.svn
web/frontend_dev.php
cache
log
stats
web/uploads

Thanks to the symfony file structure, you don't have too many files or directories to exclude manually from the synchronization. If you want to learn more about the way the files are organized in a symfony project, read the file structure chapter of the symfony book.

Production server configuration

For your project to work in the production server, the symfony framework has to be installed in the host.

Installing symfony in a production server

There are several ways to install symfony on a server, but they are not all adapted to a production environment. For instance, doing a PEAR install requires administrator rights on directories that might not be open to you if you share a web server.

Based on the principle that you will probably host several projects using symfony on the production web server, the recommended symfony installation is to uncompress the archive of the framework in a specific directory. Only the lib/ and data/ directories are necessary in a production server, so you can get rid of the other files (bin/, doc/, test/ and the files from the root directory).

You should end up with a file structure looking like:

/home/myaccount/    
  symfony/
    lib/
    data/
  askeet/
    apps/
      frontend/
    batch/
    cache/
    config/
    data/
    doc/
    lib/
    log/
    test/
    web/

For the askeet project to use the symfony classes, you have to set up two symbolic links between the application lib/symfony and data/symfony, and the related directories in the symfony installation:

$ cd /home/myaccount/askeet
$ ln -sf /home/myaccount/symfony/lib lib/symfony
$ ln -sf /home/myaccount/symfony/data data/symfony

Alternatively, if you don't have command line access, the files of the framework can be copied directly into the project lib/ and data/ directories:

copy /home/myaccount/symfony/lib/*   into /home/myaccount/askeet/lib/symfony
copy /home/myaccount/symfony/data/*  into /home/myaccount/askeet/data/symfony

Beware that in this case, each time you update the framework, you have to do it in all your projects.

For more information, all the possible ways to install symfony are described in the installation chapter of the symfony book.

Access to the symfony commands in production

While developing, you took the good habit of using:

$ symfony clear-cache

...each time you change the configuration or the object model of the project. When you upload a new version of your project in production, the cache also needs to be cleared if you want the application to work. You can easily do it by deleting the full content of the askeet/cache/ directory (by ftp or with a ssh console). Alternatively, you can have the power of the symfony command line at the price of a slightly longer installation.

To use the command line, you need to install the pake utility. Pake is a PHP tool similar to the make command. It automates some administration tasks according to a specific configuration file called pakefile.php. The symfony command line uses the pake utility, and each time you type symfony, you actually call pake with a special pakefile.php located in the symfony/bin/ directory (find more about pake in symfony in the project creation chapter of the symfony book). If you install symfony via PEAR, pake is installed as a requirement, so you usually don't see it at all and don't need to bother about what comes next. But if you do a manual installation, you have to uncompress the pake directory (get it from your symfony pear installation or download it from the pake website) into your directory in the production server. Just like for the symfony libs, you also have to add a symlink in order to enable symfony to use pake:

$ ln -sf /home/myaccount/pake/lib lib/pake

You should end up with something like this:

/home/myaccount/
  pake/
    lib/      
  symfony/
    lib/
    data/
  askeet/
    apps/
      frontend/
    batch/
    cache/
    config/
    data/
      symfony/ -> /home/myaccount/symfony/data
    doc/
    lib/
      symfony/ -> /home/myaccount/symfony/lib
      pake     -> /home/myaccount/pake/data
    log/
    test/
    web/

To call the symfony command to do a clear-cache, you need to do:

$ cd /home/myaccount/askeet/
$ php lib/pake/bin/pake.php -f lib/symfony/data/symfony/bin/pakefile.php clear-cache

Alternatively, you can create a file called symfony in the home/myaccount/askeet/ with:

#!/bin/sh

php lib/pake/bin/pake.php -f lib/symfony/data/symfony/bin/pakefile.php $@

Then, all you need to do in order to clear the cache is that good old

$ symfony clear-cache

Web command

If you want to have the power of the pake utility but without command line access, you can also create a web access for the clear-cache command.

For instance, you could save the following webpake.php in your /home/myaccount/askeet/web/ directory:

<?php
 
// as we are in the web/ dir, we need to go up one level to get to the project root
chdir(dirname(__FILE__).DIRECTORY_SEPARATOR.'..');
 
include_once('/lib/symfony/pake/bin/pake.php');
 
$pake = pakeApp::get_instance();
try
{
  $ret = $pake->run('/data/symfony/bin/pakefile.php', 'clear-cache');
}
catch (pakeException $e)
{
  print "<strong>ERROR</strong>: ".$e->getMessage();
}
 
?>

Then, clearing the cache could be done by navigating to:

http://myaskeetprodserver.com/webpake.php

Upgrading your application

There will be times in the life of your project when you need to switch between two versions of an application. It can be in order to correct bugs, or to upload new features. You can also be faced with the problem of switching between two versions of the database. If you follow a few good practices, these actions will prove easy and harmless.

Show unavailability notice

Between the moment when you start the data transfer and the moment you clear the cache (if you modify the configuration or the data model), there are sometimes more than a few seconds of delay. You must plan to display an unavailability notice for users trying to browse the site at that very moment.

In the application settings.yml, define the unavailable_module and unavailable_action settings:

all:
  .settings:
    unavailable_module:     content
    unavailable_action:     unavailable

Create an empty content/unavailable action and a unavailableSuccess.php template:

// askeet/apps/frontend/modules/content/actions/actions.class.php
public function executeUnavailable()
{
  $this->setTitle('askeet! » maintenance');
}
 
// askeet/apps/frontend/modules/content/templates/unavailableSuccess.php
<h1>Askeet: Site maintenance</h1>
 
<p>The askeet website is currently being updated.</p>
 
<p>Please try again in a few minutes.</p>
 
<p><i>The askeet team</i></p>

Now each time that you want to make your application unavailable, just change the available setting:

all:

  .settings:

    available:              off

Don't forget that for a configuration change to be taken into account in production, you need to clear the cache.

Use two versions of your application

A good way to avoid unavailability is to have the project root folder configured as a symlink. For instance, imagine that you are currently using the version 123 of your application, and that you want to switch to the version 134. If your web server root is set to /home/myaccount/askeet/web/ and that the production folder looks like that:

/home/myaccount/
  pake/
    lib/      
  symfony/
    lib/
    data/
  askeet/     -> /home/production/askeet.123/
  askeet.123/
  askeet.134/

Then you can instantly switch between the two versions by changing the symlink:

$ ln -sf /home/myaccount/askeet/ /home/myaccount/askeet.134/

The users will see no interruption, and yet all the files used after the change of the symlink will be the ones of the new release. If, in addition, you emptied the cache/ folder of your release 134, you don't even need to launch a clear-cache after switching applications.

Switching databases

You can extrapolate that technique to switching databases. Remember that the address of the database used by your application is defined in the databases.yml configuration file. If you create a copy of the database with a new name, say askeet.134, you just need to write in the askeet.134/apps/frontend/config/databases.yml:

all:
  propel:
    class:          sfPropelDatabase
    param:
      phptype:      mysql
      hostspec:     localhost
      database:     askeet.134
      username:     myuser
      password:     mypassword
      compat_assoc_lower:  true
      compat_rtrim_string: true

As the databases.yml will be switched as the same time as the application itself, your askeet will instantly start querying the new database.

This technique is especially useful if your application has a big traffic and if you can't afford any service interruption.

See you Tomorrow

Synchronization is often a big issue for high traffic websites, but thanks to the file structure of the symfony projects, it should not create any problem for askeet.

Tomorrow, we will talk about the way to adapt askeet to other languages. The patient speakers call it internationalization, the others find it more convenient to talk about i18n. Symfony has built-in support for multilingual sites, so that should not be a big deal.

You can still post your questions and suggestions in the askeet forum. And did you try to ask one in the brand new askeet website?