View source for BOA Server/Development workflow

This page describes how to build, test, release, manage, maintain and update the Transition Network Drupal websites.

'''Before continuing, please make sure you understand [[BOA Server]] and have followed and read about BOA, Platforms, Makefiles and more. You must also understand [[BOA_Server#Platform_environments|where to do your work and testing]]!'''

== Development (locally on your system) ==

=== Kinds of Drupal development: Code vs. features ===
The kind of development you'll be doing will dictate where and how to do it:
* '''Writing code, modules or themes''' -- You should be doing this on a LOCAL copy of the site on your OWN machine, then committing the work when features are completed or things need to be tested/reviewed... See ''Drupal code development'', below.
* '''Drupal UI''':  Altering Drupal configuration via Views, Panels etc -- If you are creating views, editing displays and altering pre-existing Drupal configuration then you can probably do this directly on STG or even PROD -- IF you have permission, and have a clearly defined remit... See ''Altering Drupal configuration through the UI'', below.
* Or '''both'''! -- Most 'proper' development work will require elements of each of the above. In this case get your own copy of the site to work with on your machine, and be prepared to use the Features module to move configuration though the environments. Learn about  [http://drupal.org/project/features Features module] now if you don't know how it works!

=== Drupal code development ===
This assumes you will be creating or editing modules and themes. You need your own copy of the site for this!

==== Setting up your own computer for development ====
Depending on your own setup, you might already have the required web development environments set up. If so, great! If not, you will need to stop and research how to do this best for your system. There are two basic approaches:
* Add the necessary software to your machine, including the web server, database server, editing software etc.
* Use virtual machine containing a pre-made development environment. There are two sub-options:
** Use [[http://drupal.org/project/quickstart Quickstart]] or [[http://drupal.org/project/drupalpro DrupalPro]] VM images with all you need to get going.
** Use BOA + your own machine: BOA now installs to a local machine, so you can get a Ubuntu VM and set it up there... Instructions at http://drupal.org/project/barracuda. That covers the web server, but you'll need to set up your system to be able to edit files inside the VM, or add needed software in there.

==== Copying a site for local development  ====
Once you're set up, you need to get the database and code. You can usually ignore the files unless there's something specific you need.
* '''Get the code''' by Using Drush Make - follow these instructions: [[BOA_Server/Building_platforms#Using_Drush_Make]]
* '''Get the database''' by using Backup & Migrate (module should be enabled on all sites) to download from the site you need to clone. Use the mysql command on the commandline to put the SQL dump file into your database - look this up if not sure! The command looks like this (replace things in [square brackets] with proper values):
<pre>mysql -u[username] -p[password] [local database name] < [path to your downloaded dump file]/[filename.mysql]</pre> Remember to unzip/untar the file first!
* Also MAKE SURE YOU DO NOT HAVE AN EMAIL RELAY RUNNING! This means you do NOT want your system sending spamming real users whilst you're working! That would be '''BAD'''. So disable/uninstall the email gateway, or set it to definitely reroute them all back to a test account under your control. Make sure you set up reroute_email if you want to keep your local mail gateway running -- see below. <pre>drush dis -y notifictions messaging piwik google_analytics mailchimp captcha mollom search</pre>
* Once the site is working locally, disable some modules using Drush that may cause issues locally:

<pre>drush dis -y session443</pre>
* Then you can enable some modules you might need to help you develop:
<pre>drush en -y devel reroute_email views_ui context_ui rules_ui</pre>

==== Editing & committing code & features ====

You should be using Git to add and commit your code to the appropriate repository (see next section on how to get this ready). If you don't know about Git, you need to learn before you continue! Here are some handy links:

'''Git links:'''
* [http://drupal.org/documentation/git Drupal Git documentation at drupal.org]
* [http://gitref.org/ Git reference at gitref.org]

==== Replacing a module/theme folder with the version from Git ====
So by now you've got the site setup, and you're working away. Assuming you used Drush Make or Aegir to build the makefile, you should have all the custom Transition modules and themes you want to work on on your local machine. However, you will not have all the Git repository files to allow you to push and pull your work around. In this case you will need to replace the Drush-downloaded module directory with one you've manually pulled from Git. 

The simplest way is to remove the existing module, then replace it with a <code>git clone</code>d one... This example assumes you will be working with a module called "my_module":
<pre>
cd /your/website/folder/drupal/docroot
cd sites/all/modules/custom
rm -R my_module
git clone git@github.com:transitionnetwork/my_module.git
cd my_module
</pre>
'''NOTES'''
* '''You should NOT be doing this in production!''' If you need to do something similar in PROD, please read the [[BOA_Server/Development_workflow#Simple_bug_fixes]] approach to update code. I would add that there ''are'' rare circumstances where this may be required in PROD -- In this case make sure you clone the repo somewhere else, then move delete old and move the new directory in one operation, like so: <code>rm -R my_module; mv /wherever/you/cloned/it/to/my_module .</code>
* Obviously make sure there is a Github repository for 'my_module' before deleting, and double check the GitHib url for this module.

=== Altering Drupal configuration through the UI ===
Much of the changes needed in a project are 'site building', and best done through the Drupal UI using various contributed modules. Most of these changes should be exported to STG and PROD via Features (see below), but some configuration cannot and you will need to make notes on how to reproduce the changes in STG and PROD.

==== Using Features ====
Once you've done your changes it becomes necessary to
@TODO

I'll add to this wiki page as I use features in my work flow [paulbooker]

==== Applying updates & feature reverts ====

== Updating platforms & sites using Aegir ==

=== Configuration changes and testing in STG ===
# Follow the instructions on how to build the latest version of the STG platform using the updated makefile: [[BOA_Server/Building_platforms]] -- make sure you label the platform appropriately, e.g. "Transition Network D6 S123"
# Depending on the kinds of changes, you will either need to create a new copy of the site, or just migrate the STG site onto the new STG platform.
## Simpler changes that won't disrupt others can be done on the '''primary Staging/Testing site''' on STG:
##* See [[BOA_Server/Development_workflow#Altering_Drupal_configuration_through_the_UI]]
##* Make any Drupal configuration changes via the UI (e.g. Views, Panels, CCK etc). Export theses changes via Features when done -- see [[BOA_Server/Development_workflow#Using_Features]] for more.
## If your changes are big, fundamental, or will stop other developers working, you must '''Clone''' the STG site into the new platform using these ''two steps'' (don't take a shortcut!):
##* Find the test site on the current STG platform in Aegir, click 'clone', change the site url ONLY. Label it properly so it's clearly a test -- e.g. 'jktest2013.puffin.webarch.net' -- all dev/test sites need to be a subdomain of '<code>puffin.webarch.net</code>'.
##* Once the clone is complete, you will need to '''Migrate''' your new test site to the new STG platform.
# Test the changes!!!:
## If testing throws up something, track down the cause and be prepared to roll back or patch the module update and add an exception to the makefile.
## Repeat the clone and test steps above until the issue is resolved.
## Assuming the testing is ok, you can '''migrate''' the STG site onto the new STG platform
# Do another quick test/sanity check.
# Delete your cloned test site version (NOT the main STG site!), if you won't need it again. If your clone test site is going to be around for a while, you will need to update the robots.txt file for the site - so that Google does not index the site: 

<code>          User-agent: *</code>

<code>          Disallow: /</code>

=== Updating sites in PROD  ===
# You need to build a PROD version of the platform using the approach in [[BOA_Server/Building_platforms]], making sure you follow the naming convention.
# If you've done your testing in STG, carry on -- if not STOP and do the 'TEST THE CHANGES' steps above!
# '''Migrate''' the www.transitionnetwork.org and news.transitionnetwork.org sites to the new PROD platform. There will a few minutes of downtime, so do this out of hours.

=== Cloning PROD to STG ===
Occasionally it might be necessary to make a fresh STG copy of a PROD site. To do so:
# 'Clone' the site in Aegir to the new domain on the SAME PROD platform -- e.g. clone www.transitionnetork.org to stg.transitionnetork.org
# 'Migrate' the new STG site to the STG platform you've built (see [[BOA_Server/Building_platforms]].
# Once the active Aegir task says 'Clone [your-PROD-site]' (after verify), you will need to change the symlinked Drupal files folder to the correct location for STG: by doing something like <code>rm files; ln -s /data/disk/tn/static/sites/transitionnetwork.org-STG/files files</code> in the appropriate ~/static/platform/sites folder (this will delete the current symlinked files reference and create a new one pointing at the correct location.)
# EITHER: A) Copy the local.settings.php from another STG site, OR; B) Edit (as Root) the sites/[your-STG-site]/local.settings.php and change the <code>puffin_server_override_settings_set_environment('Production');</code> line to have 'Staging' instead of 'Production' -- this ensures emails are redirected, and the Environment Indicator is set appropriately.
# 'Edit' the site in Aegir and add dev.* alias -- e.g. dev.stg..transitionnetork.org -- this allows easier development and cache-free views of pages.
# Finally and VERY IMPORTANT: '''disable the dangerous/email sending modules''' per [[BOA_Server/Development_workflow#Copying_a_site_for_local_development]] (<pre>drush dis -y notifictions messaging piwik google_analytics mailchimp captcha mollom search</pre>)

== Module & core updates for sites ==
You will be basically building the platform again with the newer modules, then migrating first the STG site, then the PROD site when testing is completed. We assume you have cloned the makefile repository and are happy using Git. If not, STOP and read about this!

The process is:
==== 1. Check the updates ==== 
# Review the module statuses on [https://www.transitionnetwork.org/admin/reports/updates www.transitionnetwork.org -> Reports -> Available updates]. NOTE that some modules SHOULD NOT be updated without great care, and these have been marked as 'update ignored' with a administrator comment. Remember to update the page with the 'Check manually' link at the top of the updates page if the 'Last checked' date is more than a day ago.
# Read the release notes with all pending module updates to be sure you know what is will happen.
==== 2. Update makefile ==== 
Review the modules that need updating in the [https://github.com/transitionnetwork/transitionnetwork.org-d6.profile/blob/master/transitionnetwork.org-d6.make Transition Network D6 makefile]. Some modules are pinned to specific versions, and others are set to use the latest recommended release; If the module is not in the "Modules - Contrib with versions or patched" section of the makefile, it will automatically be updated when you build the platform.

If the module ''is'' in the "Modules - Contrib with versions or patched" of the makefile, it is pinned to a version, patched, or using 'Dev' release:
# Always read the explanation in the makefile and decide if the update still makes sense, is worth the risk, and fixes the issue.
# In the case of modules pinned to a set version (e.g. 2.4), be careful, read the release notes and expect to need to do extra testing.
# In the case of modules set to use the latest 'Dev' version (e.g. 1.x-dev):
## Firstly check to see if there is a new proper release version of the module ''that is newer than the Dev version in use currently''. If so, comment out the section for module in the 'versions or patched' section, and add it to the later 'Modules - Contrib' section (respecting the alphabetical order). You will need to test this thouroughly.
## If no proper release is available, the module will simply but updated to the latest 'dev' when the platform is built. This '''could'' bring bugs, so test!
# Sometimes a module is patched. If so, check to see if the new proper release or dev version includes the patch. If yes, comment out the patch line in the makefile. If no, be aware the patch may not apply any more -- Aegir will notify you of this in its logs when the platform fails to build.

==== 3. Commit changes ====
# Each time you alter the makefile you should commit with a message - e.g. <code>git add -A; git commit -m "moved tagadelic from DEV to 2.3"</code>
# Once you're done with all the module updates, push the makefile (if editing locally rather than on GitHub.)

==== 4. TEST the changes in STG ====
Do this: [[BOA_Server/Development_workflow#Configuration_changes_and_testing_in_STG]]

==== 5. Release to PROD ====
# You need to build a PROD version of the platform using the approach in [[BOA_Server/Building_platforms]], making sure you follow the naming convention.
# If you've done your testing in STG, carry on -- if not STOP and do the 'TEST THE CHANGES' steps above!
# '''Migrate''' the www.transitionnetwork.org and news.transitionnetwork.org sites to the new PROD platform. There will a few minutes of downtime, so do this out of hours.

=== Rolling back a failed migration ===
'''@TODO CHECK IF THIS PROCESS IS CORRECT'''

If there's a problem '''and only a short time (hours at most) has gone by''', you can disable the current site in PROD, then restore the backup that was made just before the migration onto the the older platform.

== Simple bug fixes in PROD ==
@TODO

== Creating a site on a platform ==


=== New sites ===
Depending on the site, you can probably just 'create' a new site using Aegir. If the site's files folder will likely end up big (> 0.5Gb), you will need to do some extra work to ensure the safety of the PROD systems -- see ''Setting up a symlink to the site folder in static'', below.

=== Cloned sites ===
When cloning a site you need to be aware of the way the files folder is set up...

In order to avoid BOA backing up HUGE sites/[sitename]/files folders, we symlink from the sites folder (inside the appropriate platform) to the <code>~/static/sites/transitionnetwork.org-[ENVIRONMENT]/files</code> folder. This usually only affects STG since DEV will not need the full set of files to work with.

'''Note''' this has already been done with www.transitionnetwork.org.

==== Setting up a symlink to the site folder in static ====
The process is:
* Log in, switch to the 'tn' user with <code>totn</code> (you might need to be root first -- this probaly needs fixing!)
* Create (or copy) the appropriate files folder in the <code>~/static/sites</code>:
<pre>
cd ~/static/sites
cp -af site.org-folder-PROD site.org-folder-STG
</pre>
* Build the D6 platform for STG
* Clone the PROD site into the new STG
* Once a STG site has been created, replace the <code>sites/[sitename]/files</code> folder with the symlink:
<pre>
cd ~/static/transition-network-d6-S001/sites/www.transitionnetwork.org
mv files files-dir
ln -s ~/static/sites/transitionnetwork.org-STG/files/ .
</pre>