I can haz staging ur drupleh content uzing teh Deployment (basement cat)?

With Drupal 7, drupal.org re-design, and all the "awesomeness" and hot source and the full pantheon of mythical gods on our web application development horizons, Drupal will still always suck if I can't stage content. Actually using an "awesome" Drupal web application over time (and not just developing it) means needing to be able to cleanly export content from one site, with its references, assets and dependencies, and import it into another. Because you can't be editing a live production site, even with as few as a few hundred simultaneous visitors.

Neither can you go around saying you're a "free as in beer" open source wonder child if you have to be a database management genius to arrange for database clusters and database migration (a la whitehouse.gov) solutions for this; seeking the "up" market (large corporations) instead of the "down" market (working stiffs, small and medium businesses, NGO's), like WordPress does, is all very well, but if you have to be a rocket scientist to cleanly move content from one site to another in order to stage content, then maybe we ought to be more honest about the real costs involved in using Drupal these days.

But, enter stage left the Deployment module! It was designed to solve this problem, and to the extent that that is true, then why wouldn't I develop all my sites using it? Otherwise:

"Ceiling cat is watching you perform search and replace node reference id's on the output of node export module content before importing to live site"

This article takes a look at the architecture of the Deployment module, sets up an initial test bed, drafts an initial acceptance test, runs the test and reports back.

Posted on: Last updated on: Written by:

With Drupal 7, drupal.org re-design, and all the “awesomeness” and hot source and the full pantheon of mythical gods on our web application development horizons, Drupal will still always suck if I can’t stage content. Actually using an “awesome” Drupal web application over time (and not just developing it) means needing to be able to cleanly export content from one site, with its references, assets and dependencies, and import it into another. Because you can’t be editing a live production site, even with as few as a few hundred simultaneous visitors.

Neither can you go around saying you’re a “free as in beer” open source wonder child if you have to be a database management genius to arrange for database clusters and database migration (a la whitehouse.gov) solutions for this; seeking the “up” market (large corporations) instead of the “down” market (working stiffs, small and medium businesses, NGO’s), like WordPress does, is all very well, but if you have to be a rocket scientist to cleanly move content from one site to another in order to stage content, then maybe we ought to be more honest about the real costs involved in using Drupal these days.

But, enter stage left the Deployment module! It was designed to solve this problem, and to the extent that that is true, then why wouldn’t I develop all my sites using it? Otherwise:

“Ceiling cat is watching you perform search and replace node reference id’s on the output of node export module content before importing to live site”

This article takes a look at the architecture of the Deployment module, sets up an initial test bed, drafts an initial acceptance test, runs the test and reports back.

Leaving to one side the fact that the module needs to be completely re-written for Drupal 7 due to new database abstractions and fields in core (meaning it may not exist for a long long time, if ever; and the abandonment of the non-functioning Drupal 5 release upon transition to Drupal 6 does not bode well either), it does promise to bring Drupal into minimum requirements for real “enterprise department” readiness (i.e. can actually be used by real people with sites in a minimal “kill the webmaster,” or at least “kill the rocket scientist” mode):

“The deployment framework is a series of modules which are designed to allow developers to easily stage Drupal data from one site to another. This includes content (nodes, taxonomy, users, etc.) as well as configuration (views, content types, system settings, etc.). Not only can it push new content, it can also push updates to existing content. Deploy automatically manages dependencies between nodes (IE nodereferences) as well as between other objects. It is designed to have a rich API which can be easily extended to be used in a variety of situations.”

Basic Deployment module architecture

The Services Module is employed to establish a pattern of web services servers and clients capable of shunting data around from one site to another. In order to overcome Drupal’s “dorm blog” mentality of auto-incrementing node id’s and using non-unique integers for indexing everything (why wasn’t this ever made the priority for some new “awesome” Drupal release?), something immediately recognizable by anyone with computer science training is employed: Universally unique identifiers (UUID), so that entity id references will never be broken. In other words, if I have a content type “gallery” pointed to by several child “photos” via node reference, they will each be given UUID rock solid never to be repeated id’s, so if I grab them and shunt them over to another site, all the node references should survive the transplant. Cool! (More recently there has been a move towards using unique “machine names” for stuff rather than integers; which might be an alternative solution at some point).

Initial test bed

Following the installation and usage documentation for the Deployment module I set up two squeaky clean source and destination servers (latest dev version, right…). I could have made a simple drush makefile, but I just banged out the following on the command line (atomic clock! sync! ntp; yes!):

$ drush dl drupal
Project drupal (6.19) downloaded to /home/dojobaires/drupal-6.19/.                                                       [success]
$ mv drupal-6.19/ deploydest
$ cd deploydest
$ drush dl deploy services
Project deploy (6.x-1.x-dev) downloaded to /home/dojobaires/deploydest/sites/all/modules/deploy.                         [success]
Project services (6.x-2.2) downloaded to /home/dojobaires/deploydest/sites/all/modules/services.                         [success]

Installed on one and cloned to the other (last time?), by doing:

$ drush sql-dump > db.sql
$ tar cvzf ../deploy.tgz .

(last time?) then grabbing deploy.tgz and getting it going as deploysource. I then followed the instructions for enabling and configuring deploy on both source and destination sites.

Source site (Staging)

I enabled these modules:

  • Deployment
  • Node Deployment
  • User Deployment
  • UUID
  • Key Authentication (Services also, a dependency)

I went to /admin/build/deploy/servers and clicked on Add a new server. I entered “Live Server” in the Name field, http://deploydest.example.com/services/xmlrpc in the URL field, left “Session id” selected in the Authentication type field and clicked the Save Deployment Server button. OK, got my destination server on.

Destination site (Production)

I enabled these modules:

  • UUID
  • Services
  • Key Authentication (Services also, a dependency)
  • XMLRPC Server
  • Node Service
  • System Service
  • User Service
  • UUID Service

I went to /admin/build/services/settings and selected “Key authentication” from the Authentication module drop down list. I made sure “Use keys” was unselected (currently unsupported) and left “Use sessid” checked. I clicked on the Save settings button.

I skipped permissions settings for now and just let things ride with admin on both source and destination sites.

I then made sure my setup was OK by carrying out the elementary story deploy outlined in Basic Usage of Deploy.

Source Site (Staging)

  • I created a new story as is, with title and body and saved the story.
  • I then went to Administer > Content management > Content (./admin/content/node), selected my story, selected Deploy from the Update options, and clicked the update button.
  • This took me to the Deploy page, at ./node_operations/deploy_now/2 (ending with whatever nid was selected) .  I selected the Live Server (destination server) and then the Username and Password fields appeared. After entering the username and password I clicked on the Submit button.
  • The deployment process went through its two phases, “Processing deployment plan” and “Pushing deployment plan” and took me to Deployment Log Details with Success in the Result column (Using the default (unlisted) “Node now” deployment plan, since I had not created any deployment plans).
  • I went to the destination site, went to the front page and there was my story!
  • Back on the staging server (source site) I created a second story, with input format Full Html, text including some <p>’s and some <li>’s (easy using http://www.fillerati.com/ — very cool), configuring it specifically not to be promoted to the front page, with a new revision (“Initial version”).
  • I then went to Administer > Content management > Content (./admin/content/node), selected the second story, selected Deploy from the Update options, and clicked the update button.
  • After deployment success, as described above, I went to the destination server, and on the front page, the second story was not listed (as per my configuration!). I went to ./admin/content/node and there it was listed. Editing the node, I saw that all my configuration had been pushed over successfully, including the revision. (Even after creating a story, giving it a few revisions, and then deploying, only the most recent revision was deployed).

UUID working behind the scenes

If you take a look at the table node_uuid you will see an entry for each node, and the uuid (in column uuid) will be used for the various node operations instead of nid in the node table. Similar tables exist for files, users, comments, taxonomy (terms and vocabularies), and files.

Acceptance test (and results)

  • Set up a squeaky clean source server, configured for deploy.
  • Set up a squeaky clean destination server, configured for deploy.
  • Confirm that basic node deployment works.
  • On the source server, get the Views Gallery module working.
    • From the module project page: “The module creates the content types, sets up the imagecache presets, creates the default gallery view that is embedded into the gallery nodes by Views Attach, and adds css to make it all look nice.”
    • Install and enable module dependencies

$ drush dl cck views views_attach nodereference_url filefield imagefield imageapi imagecache token custom_pagers lightbox2 views_slideshow

Project cck (6.x-2.8) downloaded to /home/dojobaires/deploysource/sites/all/modules/cck. [success]

Project views (6.x-2.11) downloaded to /home/dojobaires/deploysource/sites/all/modules/views. [success]

Project views_attach (6.x-2.2) downloaded to /home/dojobaires/deploysource/sites/all/modules/views_attach. [success]

Project nodereference_url (6.x-1.6) downloaded to /home/dojobaires/deploysource/sites/all/modules/nodereference_url. [success]

Project filefield (6.x-3.7) downloaded to /home/dojobaires/deploysource/sites/all/modules/filefield. [success]

Project imagefield (6.x-3.7) downloaded to /home/dojobaires/deploysource/sites/all/modules/imagefield. [success]

Project imageapi (6.x-1.9) downloaded to /home/dojobaires/deploysource/sites/all/modules/imageapi. [success]

Project imagecache (6.x-2.0-beta10) downloaded to /home/dojobaires/deploysource/sites/all/modules/imagecache. [success]

Project token (6.x-1.15) downloaded to /home/dojobaires/deploysource/sites/all/modules/token. [success]

Project custom_pagers (6.x-1.0-beta2) downloaded to /home/dojobaires/deploysource/sites/all/modules/custom_pagers. [success]

Project lightbox2 (6.x-1.11) downloaded to /home/dojobaires/deploysource/sites/all/modules/lightbox2. [success]

Project views_slideshow (6.x-2.3) downloaded to /home/dojobaires/deploysource/sites/all/modules/views_slideshow. [success]

  • Configure everything just right, making sure that we have at least one gallery and some photos in it.

After enabling the modules (content: everything except the pesky field permissions; imagecache: imageapi, imageapi gd2 (either gd2 or image magick toolkit), imagecache, imagecache ui; custom pagers, lightbox 2, token, views: everything), download and enable the Views Gallery module.

$ drush dl views_gallery
Project views_gallery (6.x-1.3) downloaded to /home/dojobaires/deploysource/sites/all/modules/views_gallery.             [success]

Enabling:

The content type Gallery has been added.
A new Views Gallery Gallery has been created for you. Go there to add photos to that gallery, or create new galleries at Add Gallery
The content type Image has been added.
The field Image (field_gallery_image) was added to the content type Image.
The field Gallery (field_gallery) was added to the content type Image.
The configuration options have been saved.

Create a gallery and add some images to it, check that everything is working (at http://deploysource.webproworkshop.com/node/5 ):

 

Views Gallery on staging server

Views Gallery on staging server

  • On the destination (live site) server
    • Enable the views gallery module and all its dependencies as on the staging server, without creating any actual content. This obviously could be done via features! Or, using the deployment module for deploying content types, views… (image cache presets???). But let’s not overcomplicate things on this run. Here we just want to know if nodereference deployment works without breaking 🙂
    • Enable the File Service module
  • On the source (staging) server
    • Enable the following modules
      • Deploy Content Type (otherwise won’t pick up on nodereference fields)
      • Deploy Files
      • Deploy Nodereferences
      • Deploy System Settings
      • UUID Service (otherwise you get “Page not found” error)
    • Create a deployment plan (as explained in Basic Usage of Deploy)
      • Go to Administer > Site building > Deployment 
      • Enter a unique name and an optional description and hit the Save deployment plan button.
      • Go to Administer > Content management > Content and select the Gallery together with the Image nodes. Select Add to deployment plan from the Update options drop down list and click on the Update button.
      • You are taken to ./node_operations/deploy/9,8,7,6,5 (list of nids), Deploy. Select the deployment plan you created earlier, to which you wish to add the content you selected, and press the Submit button.
      • You are taken to a list of items, and given the chance to remove items. Do nothing unless you want to on this page.
      • Go to Administer > Content management > Content types and click the Deploy tab. Select the same deployment plan and the two content types associated with the content, Gallery and Image. Click on the Add content type button.
      • Go to Administer > Site building > Deployment (admin/build/deploy) and click the push link corresponding to the deployment plan you have just associated content with, to push it to the live sit. You will be prompted for the destination server username and password and the process will follow as with a simple node now push.
      • #fail Because http://drupal.org/node/418910 Nodereferences do not deploy properly unless using autocomplete widget
      • Stop test run.
  • Deploy the gallery and related image nodes to the destination (live site) server.
  • On the destination (live) server, check to see if all the dependencies (content types, views) have been met.
  • Check to see if the node references are correctly followed (i.e. if the images appear correctly in the gallery they were pointing to on the staging server).

Running a simpler example

  • Create a content type called article, and another called author, each with basic “story”-like title and body, except that the article will have a nodereference (using autocomplete widget!!!) to author.
  • Create Article one with author Author one, Article two with author Author two.
  • Create deployment plan, add the two content types, add the two articles and authors nodes.
  • Push
  • Success! Both content types and content have been deployed! “Author two” has nid 12 on source, and nid 8 on destination, but the links do not break, nodereference deployment has worked! 

Simple deploy example with nodereferenceSimple deploy example with nodereference

Success!

Conclusions and road map

The gotchas with this evolving module are very finite in number, but may be prohibitive on certain projects. Some of the gotchas are mentioned in Troubleshooting common deploy problems, but not all.

References

  1. Deployment module
  2. Installation and usage documentation 
  3. Basic Usage of Deploy 
  4. UUID
  5. Services Module
  6. Node export module
  7. Views Gallery 
  8. Troubleshooting common deploy problems
  9. Nodereferences do not deploy properly unless using autocomplete widget (problem with services, not with deploy; however…)