BOA Server/Development workflow

This page describes how to build, test, release, manage, maintain and update the Transition Network Drupal websites.

Before continuing, please make sure you understand BOA Server and have followed and read about BOA, Platforms, Makefiles and more. You must also understand where to do your work and testing!

Development (locally on your system)

Kinds of Drupal development: Code vs. features

The kind of development you'll be doing will dictate where and how to do it:

Writing code, modules or themes -- You should be doing this on a LOCAL copy of the site on your OWN machine, then committing the work when features are completed or things need to be tested/reviewed... See Drupal code development, below.
Drupal UI: Altering Drupal configuration via Views, Panels etc -- If you are creating views, editing displays and altering pre-existing Drupal configuration then you can probably do this directly on STG or even PROD -- IF you have permission, and have a clearly defined remit... See Altering Drupal configuration through the UI, below.
Or both! -- Most 'proper' development work will require elements of each of the above. In this case get your own copy of the site to work with on your machine, and be prepared to use the Features module to move configuration though the environments. Learn about Features module now if you don't know how it works!

Drupal code development

This assumes you will be creating or editing modules and themes. You need your own copy of the site for this!

Setting up your own computer for development

Depending on your own setup, you might already have the required web development environments set up. If so, great! If not, you will need to stop and research how to do this best for your system. There are two basic approaches:

Add the necessary software to your machine, including the web server, database server, editing software etc.
Use virtual machine containing a pre-made development environment. There are two sub-options:
- Use [Quickstart] or [DrupalPro] VM images with all you need to get going.
- Use BOA + your own machine: BOA now installs to a local machine, so you can get a Ubuntu VM and set it up there... Instructions at http://drupal.org/project/barracuda. That covers the web server, but you'll need to set up your system to be able to edit files inside the VM, or add needed software in there.

Copying a site for local development

Once you're set up, you need to get the database and code. You can usually ignore the files unless there's something specific you need.

Get the code by Using Drush Make - follow these instructions: BOA_Server/Building_platforms#Using_Drush_Make
Get the database by using Backup & Migrate (module should be enabled on all sites) to download from the site you need to clone. Use the mysql command on the commandline to put the SQL dump file into your database - look this up if not sure! The command looks like this (replace things in [square brackets] with proper values):

mysql -u[username] -p[password] [local database name] < [path to your downloaded dump file]/[filename.mysql]

Remember to unzip/untar the file first!

Also MAKE SURE YOU DO NOT HAVE AN EMAIL RELAY RUNNING! This means you do NOT want your system sending spamming real users whilst you're working! That would be BAD. So disable/uninstall the email gateway, or set it to definitely reroute them all back to a test account under your control. Make sure you set up reroute_email if you want to keep your local mail gateway running -- see below.
```
drush dis -y notifictions messaging piwik google_analytics mailchimp captcha mollom search
```
Once the site is working locally, disable some modules using Drush that may cause issues locally:

drush dis -y session443

Then you can enable some modules you might need to help you develop:

drush en -y devel reroute_email views_ui context_ui rules_ui

Editing & committing code & features

You should be using Git to add and commit your code to the appropriate repository (see next section on how to get this ready). If you don't know about Git, you need to learn before you continue! Here are some handy links:

Git links:

Replacing a module/theme folder with the version from Git

So by now you've got the site setup, and you're working away. Assuming you used Drush Make or Aegir to build the makefile, you should have all the custom Transition modules and themes you want to work on on your local machine. However, you will not have all the Git repository files to allow you to push and pull your work around. In this case you will need to replace the Drush-downloaded module directory with one you've manually pulled from Git.

The simplest way is to remove the existing module, then replace it with a git cloned one... This example assumes you will be working with a module called "my_module":

cd /your/website/folder/drupal/docroot
cd sites/all/modules/custom
rm -R my_module
git clone git@github.com:transitionnetwork/my_module.git
cd my_module

NOTES

You should NOT be doing this in production! If you need to do something similar in PROD, please read the BOA_Server/Development_workflow#Simple_bug_fixes approach to update code. I would add that there are rare circumstances where this may be required in PROD -- In this case make sure you clone the repo somewhere else, then move delete old and move the new directory in one operation, like so: rm -R my_module; mv /wherever/you/cloned/it/to/my_module .
Obviously make sure there is a Github repository for 'my_module' before deleting, and double check the GitHib url for this module.

Altering Drupal configuration through the UI

Much of the changes needed in a project are 'site building', and best done through the Drupal UI using various contributed modules. Most of these changes should be exported to STG and PROD via Features (see below), but some configuration cannot and you will need to make notes on how to reproduce the changes in STG and PROD.

Using Features

Once you've done your changes it becomes necessary to @TODO

I'll add to this wiki page as I use features in my work flow [paulbooker]

Applying updates & feature reverts

Updating platforms & sites using Aegir

Configuration changes and testing in STG

Follow the instructions on how to build the latest version of the STG platform using the updated makefile: BOA_Server/Building_platforms -- make sure you label the platform appropriately, e.g. "Transition Network D6 S123"
Depending on the kinds of changes, you will either need to create a new copy of the site, or just migrate the STG site onto the new STG platform.
1. Simpler changes that won't disrupt others can be done on the primary Staging/Testing site on STG:
  - See BOA_Server/Development_workflow#Altering_Drupal_configuration_through_the_UI
  - Make any Drupal configuration changes via the UI (e.g. Views, Panels, CCK etc). Export theses changes via Features when done -- see BOA_Server/Development_workflow#Using_Features for more.
2. If your changes are big, fundamental, or will stop other developers working, you must Clone the STG site into the new platform using these two steps (don't take a shortcut!):
  - Find the test site on the current STG platform in Aegir, click 'clone', change the site url ONLY. Label it properly so it's clearly a test -- e.g. 'jktest2013.puffin.webarch.net' -- all dev/test sites need to be a subdomain of 'puffin.webarch.net'.
  - Once the clone is complete, you will need to Migrate your new test site to the new STG platform.
Test the changes!!!:
1. If testing throws up something, track down the cause and be prepared to roll back or patch the module update and add an exception to the makefile.
2. Repeat the clone and test steps above until the issue is resolved.
3. Assuming the testing is ok, you can migrate the STG site onto the new STG platform
Do another quick test/sanity check.
Delete your cloned test site version (NOT the main STG site!), if you won't need it again. If your clone test site is going to be around for a while, you will need to update the robots.txt file for the site - so that Google does not index the site:

User-agent: *

Disallow: /

Updating sites in PROD

You need to build a PROD version of the platform using the approach in BOA_Server/Building_platforms, making sure you follow the naming convention.
If you've done your testing in STG, carry on -- if not STOP and do the 'TEST THE CHANGES' steps above!
Migrate the www.transitionnetwork.org and news.transitionnetwork.org sites to the new PROD platform. There will a few minutes of downtime, so do this out of hours.

Cloning PROD to STG

Occasionally it might be necessary to make a fresh STG copy of a PROD site. To do so:

'Clone' the site in Aegir to the new domain on the SAME PROD platform -- e.g. clone www.transitionnetork.org to stg.transitionnetork.org
'Migrate' the new STG site to the STG platform you've built (see BOA_Server/Building_platforms.
Once the active Aegir task says 'Clone [your-PROD-site]' (after verify), you will need to change the symlinked Drupal files folder to the correct location for STG: by doing something like rm files; ln -s /data/disk/tn/static/sites/transitionnetwork.org-STG/files files in the appropriate ~/static/platform/sites folder (this will delete the current symlinked files reference and create a new one pointing at the correct location.)
EITHER: A) Copy the local.settings.php from another STG site, OR; B) Edit (as Root) the sites/[your-STG-site]/local.settings.php and change the puffin_server_override_settings_set_environment('Production'); line to have 'Staging' instead of 'Production' -- this ensures emails are redirected, and the Environment Indicator is set appropriately.
'Edit' the site in Aegir and add dev.* alias -- e.g. dev.stg..transitionnetork.org -- this allows easier development and cache-free views of pages.
Finally and VERY IMPORTANT: disable the dangerous/email sending modules per BOA_Server/Development_workflow#Copying_a_site_for_local_development (
```
drush dis -y notifictions messaging piwik google_analytics mailchimp captcha mollom search
```
)

Module & core updates for sites

You will be basically building the platform again with the newer modules, then migrating first the STG site, then the PROD site when testing is completed. We assume you have cloned the makefile repository and are happy using Git. If not, STOP and read about this!

The process is:

1. Check the updates

Review the module statuses on www.transitionnetwork.org -> Reports -> Available updates. NOTE that some modules SHOULD NOT be updated without great care, and these have been marked as 'update ignored' with a administrator comment. Remember to update the page with the 'Check manually' link at the top of the updates page if the 'Last checked' date is more than a day ago.
Read the release notes with all pending module updates to be sure you know what is will happen.

2. Update makefile

Review the modules that need updating in the Transition Network D6 makefile. Some modules are pinned to specific versions, and others are set to use the latest recommended release; If the module is not in the "Modules - Contrib with versions or patched" section of the makefile, it will automatically be updated when you build the platform.

If the module is in the "Modules - Contrib with versions or patched" of the makefile, it is pinned to a version, patched, or using 'Dev' release:

Always read the explanation in the makefile and decide if the update still makes sense, is worth the risk, and fixes the issue.
In the case of modules pinned to a set version (e.g. 2.4), be careful, read the release notes and expect to need to do extra testing.
In the case of modules set to use the latest 'Dev' version (e.g. 1.x-dev):
1. Firstly check to see if there is a new proper release version of the module that is newer than the Dev version in use currently. If so, comment out the section for module in the 'versions or patched' section, and add it to the later 'Modules - Contrib' section (respecting the alphabetical order). You will need to test this thouroughly.
2. If no proper release is available, the module will simply but updated to the latest 'dev' when the platform is built. This 'could bring bugs, so test!
Sometimes a module is patched. If so, check to see if the new proper release or dev version includes the patch. If yes, comment out the patch line in the makefile. If no, be aware the patch may not apply any more -- Aegir will notify you of this in its logs when the platform fails to build.

3. Commit changes

Each time you alter the makefile you should commit with a message - e.g. git add -A; git commit -m "moved tagadelic from DEV to 2.3"
Once you're done with all the module updates, push the makefile (if editing locally rather than on GitHub.)

4. TEST the changes in STG

Do this: BOA_Server/Development_workflow#Configuration_changes_and_testing_in_STG

5. Release to PROD

You need to build a PROD version of the platform using the approach in BOA_Server/Building_platforms, making sure you follow the naming convention.
If you've done your testing in STG, carry on -- if not STOP and do the 'TEST THE CHANGES' steps above!
Migrate the www.transitionnetwork.org and news.transitionnetwork.org sites to the new PROD platform. There will a few minutes of downtime, so do this out of hours.

Rolling back a failed migration

@TODO CHECK IF THIS PROCESS IS CORRECT

If there's a problem and only a short time (hours at most) has gone by, you can disable the current site in PROD, then restore the backup that was made just before the migration onto the the older platform.

Simple bug fixes in PROD

@TODO

Creating a site on a platform

New sites

Depending on the site, you can probably just 'create' a new site using Aegir. If the site's files folder will likely end up big (> 0.5Gb), you will need to do some extra work to ensure the safety of the PROD systems -- see Setting up a symlink to the site folder in static, below.

Cloned sites

When cloning a site you need to be aware of the way the files folder is set up...

In order to avoid BOA backing up HUGE sites/[sitename]/files folders, we symlink from the sites folder (inside the appropriate platform) to the ~/static/sites/transitionnetwork.org-[ENVIRONMENT]/files folder. This usually only affects STG since DEV will not need the full set of files to work with.

Note this has already been done with www.transitionnetwork.org.

Setting up a symlink to the site folder in static