Sharing Engine/Technical Proposal

Proposed PSE Alpha Architecture

Given the weight of pros and cons for our discussions (see below for more), for maximum speed of development and flexibility during Alpha and Beta stages we have agreed a 'Monolithic' approach which re-uses the existing Project Profile content type, BUT only after it is tidied up per Sharing Engine/TN Project Form Changes. This maximises use of existing Drupal and site administration infrastructure without needed changes (outside of projects) to get what we need done. Should the PSE become wildly popular after the Alpha stages, we can either a) throw hardware at it, or b) migrate to another approach later in the project -- much of what is done will be re-usable and modular.

There might possibly be the need for a simple 'Sharing Engine Client Website' content type added to TN.Org, which would relate a participating website to an initiative and webmaster and allows all submissions to be grouped, blocked or included by Client. This would allow any given widget to be authenticated and allowed/denied to post to the PSE.

Key requirements

• Submitters do not need to go to their email to continue, because the Project Profile content type form is be merged with the user login/registration/password recovery form.
• Widget owner pre-moderates projects sourced from their site

Project Content type

We will be using the existing Project Profile content type, with changes (Sharing_Engine/TN_Project_Form_Changes) committed to a new Drupal Feature here: Sharing_Engine/Drupal_features_and_modules#Features.

The plan for the Project Profile content type is:

1. I grab the existing Project and views as a Feature, add the new fields including a 'gulag' field and a 'moderation status' field, pre-set to 'Accepted' for current content.
2. Write a little code to take the content out of the fields to be dropped and put it in the Gulag with HTML headings
3. Laura then manages the handful of verbose projects with lots in their Gulag.
4. We remove the unneed fields and eventually the gulag
5. Tweaks made to Project views (directory) to hide unmoderated ones
6. Widget fields are now just the mandatory fields from the refreshed Project CT, not a separate CT. All existing views, permissions and contact methods work for existing projects AND new ones.
7. New 'Project moderation' view showing pending submissions.

URL structure

Everything for the PSE alpha will live at www.transitionnetwork.org/pse... The sub-paths will be like this:

• pse/entry/* -- All things to add projects via the widget... Not usually exposed except via widget.
• pse/view/* -- the contents for the view widget(s) which return a responsive page with either just a nearby project list pse/view/basic and a much more complex widget with tabs for pse/view/full
• pse/widget/* -- pages for getting (pse/widget/get-code) and signing up ((pse/widget/signup)
• pse/admin/* -- pages for staff and webmasters who use the widget to manage the Project content created.

Integrated Login/Registration

We need the user data as part of the submission, which for the alpha may as well be the standard Drupal login/registration process. This step will be integrated with the entry widget.

Rather than make exceptions and fundamentally alter the way Drupal handles users registering, we should allow users to logged in & registered via Google, Facebook, Twitter etc -- lowering the barrier to entry for all.

Process & architecture overview

1. All work, submissions and display is done by existing TN.org site
2. Clean, responsive, simple theme that will be applied to all widget (remote) forms and views
3. Simple 'Project Submission' content type added to TN.Org that includes
   • Fields per the the Information Architecture
   • Mollom or reCaptcha anti-spam protection
   • Integrated newsletter signup
   • Analytics
4. Moderator/Webmaster views of Project Submissions
5. A little custom module to allow a moderated, accepted Project Submission to be converted into a full Project node with all relevant user and initiatives references added
6. Notifications to various parties on certain events TODO!

Other considerations

• Varnish behaviour for submission of forms (must be HTTPS)
• reporting, analytics and monitoring
• migration of system to Drupal 7 at some point
• how to handle abuse/flooding of widget
• Input of location/map data

Discussion: Is the PSE a separate, service-based website, or is it fully integrated into TranstionNetwork.org?

This question was discussed in US Initiative Sharing & Project Sharing Engine:Basic process and behaviour summary in the 'Satellite or monolith?' section. Based on existing Drupal 6 infrastructure and Information Architecture already defined, there are two broad approaches to this project:

1. Satellite -- PSE separate from existing TN.org website - and requiring modules and code to share users and data between two
2. Monolithic -- PSE Integral to the exising site - and needing to keep not-yet-moderated submissions separate and hidden from normal visitors

Satellite

Pros

• easy to set up;
• easy to stay focused and use best of Drupal 7 modules and themes to quickly get where we need to;
• easy to extend, port and share codebase with other groups; 
• can be on any server anywhere
• completely separate so can do exotic or complex things without complicating an already huge TN.org D6 install
• if PSE is wildly sucessful we can move it to another server very easily
• easy to create meaningful and clean API for data submission/display based on fresh D7 infrastructure

Cons

• (To begin with) it would be yet another site on an already very busy server
• most of the coding & development would be to integrate this satellite site's sources of data with users, permissions and project data on the main site
• More work to do for a basic Alpha version, and therefore would only really make sense in beta or final version

Monolithic

Pros

• fits in with existing infrastructure very well
• little extra modules or setup needed as all likely required modules are present
• all users, permissions and existing moderation structures are in place, including webmaster and TN Admin access
• no code needed to transport data between two sites, only simple code to map a sucessfully moderated submission to a fully-fledged project node
  

Cons

• makes a huge site (in terms of server space and footprint) more huge-rer
• if PSE is wildly sucessful, we would need need to expand the main server capacity OR break this work off into a separate install somewhere else
• makes it harder to create meaningful and clean API for data submission/display because of existing 'bloated' data structures and old D6 infrastructure

Discussion: Possible PSE Drupal architectures

Given the limited time and budget, the Alpha version of this project's widgets must be kept simple and use off-the-shelf components where possible. With this in mind, the broad options are:

Submissions are for existing Project content type

• existing Project Profile content type is overly complex, poorly classified (taxonomy etc)
• CT is not able to be posted by anonymous visitors
• CT will need cleaning up and extra fields added for the workflow
• This approach forces all data to live within system and structures as-is, which has both benefits for a speedy Alpha product, but possible issues later on.
• The Project Profile CT and associated directory is already being reviewed and updated.

Submissions are a new content type, then mapped to Project nodes

A small content type can be created and exposed, then anonymous users submit this CT within the widget... This is the most flexible approach by far as;

• moderation fits well into existing structures
• all users and related data is present and can be related to any kind of nodes as needed
• we can use all existing node- and content-type modules, including Location, Gmap (or similar), MailChimp etc and also do registration from the same form.
• it's heaviest for system because there could potentially be many spam posts sent and inserting/deleting these records is bad for the database's performance.
• once moderated, submission nodes are either A) have their data mapped from a submissions to a new project node; or b) the new type of lightweight project node is published and added to existing streams of data
• moderation is roughly the same as for normal node processes
• more work to extend to other submission types (initiatives, events etc)

Submissions are Webform entries, then mapped to Project nodes

A webform (instead of a content type) is added and shown in the widget...

• PSE workflows already fit standard Webform module behaviours;
• it's light weight but limited because there are only a few types fo field available for webforms (no Location etc)
• data is mapped from individual webform submissions to standard Project nodes during sucessful moderation
• moderation a more complex and separate from normal Node processes, would probably need more code written
• it can rapidly be re-used to allow any remote submission with little overhead (compared to all the Views and CT changes for nodes)