Towards a straight up Drupal development workflow for the working stiff – can’t we all just follow industry standards?

I need to write an installation profile for Project Flow and Tracker in order to publish it right on http://drupal.org so that it can be contributed to the community. My goal however is to avoid coming up with one of those monolithic install profiles whose drupal core or modules and themes you can never update on your own without breaking everything, whose functionality you have to take as is on an all or nothing basis, and which is generally speaking unmaintainable and has to be replaced en bloc (sometimes en bric) whenever there is a new release (i.e. erase everything, copy in the new release, run update.php...).

The promise of Contexts and Features driven development was that you got to develop incrementally with everything in code (modular, versionable, automatically buildable), then with the help of Drush Make and the Install Profile API  you could make a modern installation profile which simply installs the features and their dependencies, and reverts them to make sure they are properly enabled. And that this cycle can be repeated transparently as the app evolves, as part of its development workflow.

But the reality is that features are the coolest thing in town, but once you knock on that mistress' door, you are imprisoned in her magic forever; and you don't wind up with reusable self-contained components for all your pains. In this article we are going to find out why that is and come to some conclusions as to what can be done about it (under the circumstances). I will present the problem and share conclusions as to how best practices can best be approximated in Drupal app development, and will illustrate conclusions I have come to as to the most sustainable method of creating an installation profile in the context of an ongoing application project.

I touched on this recently in Features driven development -- the nitty gritty and thoughts on the future of sustainable "in code" development, but now I want to systematize my ideas and figure out what to do about it.

The problem

The most important thing to consider is that Features as development workflow breaks a fundamental law in Software Developoment:

Software development law #1: there is no one-to-one relationship between use cases (features, functional requirements, user stories, etc.) and project configuration artifacts (modules, themes, menus, blocks, panels, settings, etc.). A single project artifact will serve as part of the implementation of many use cases; a single use case will be implemented in many project artifacts.

Posted on: Last updated on: Written by:

I need to write an installation profile for Project Flow and Tracker in order to publish it right on http://drupal.org so that it can be contributed to the community. My goal however is to avoid coming up with one of those monolithic install profiles whose drupal core or modules and themes you can never update on your own without breaking everything, whose functionality you have to take as is on an all or nothing basis, and which is generally speaking unmaintainable and has to be replaced en bloc (sometimes en bric) whenever there is a new release (i.e. erase everything, copy in the new release, run update.php…).

The promise of Contexts and Features driven development was that you got to develop incrementally with everything in code (modular, versionable, automatically buildable), then with the help of Drush Make and the Install Profile API  you could make a modern installation profile which simply installs the features and their dependencies, and reverts them to make sure they are properly enabled. And that this cycle can be repeated transparently as the app evolves, as part of its development workflow.

But the reality is that features are the coolest thing in town, but once you knock on that mistress’ door, you are imprisoned in her magic forever; and you don’t wind up with reusable self-contained components for all your pains. In this article we are going to find out why that is and come to some conclusions as to what can be done about it (under the circumstances). I will present the problem and share conclusions as to how best practices can best be approximated in Drupal app development, and will illustrate conclusions I have come to as to the most sustainable method of creating an installation profile in the context of an ongoing application project.

I touched on this recently in Features driven development — the nitty gritty and thoughts on the future of sustainable “in code” development, but now I want to systematize my ideas and figure out what to do about it.

The problem

The most important thing to consider is that Features as development workflow breaks a fundamental law in Software Developoment:

Software development law #1: there is no one-to-one relationship between use cases (features, functional requirements, user stories, etc.) and project configuration artifacts (modules, themes, menus, blocks, panels, settings, etc.). A single project artifact will serve as part of the implementation of many use cases; a single use case will be implemented in many project artifacts.

Now, the definition of “feature” given at http://drupal.org/project/features is “a collection of Drupal entities which taken together satisfy a certain use-case”, implying that given another use case, there is another “feature” that can satisfy (implement, support) it. But this concept is fatally flawed in the context of any software app development, including that of Drupal apps, since if I package some functionality in a feature, any of the project artifacts contained therein are “imprisoned”: if a content type forms part of the implementation of business objects contained in Use Case 1, and they are in Feature 1, even though the same content type is germain to Use Case 2, it cannot be included in Feature 2, because it’s already in Feature 1. And the same goes for settings, permissions, etc. and things get even more complicated as the UI comes into play (blocks, contexts or panels and their contexts, panes, etc., obviously are “re-used” and exist as dependencies for multiple use cases).

This is why despite the fact that contexts, features and kits have been proposed and wildly acclaimed (by myself also) to solve the features dependencies problem (note: this highly informative article from the folks at  FunnyMonkey shows that the only (incomplete) solution is painstaking and artesanal features refactoring), the sad truth is that contexts and features becomes just another “wall of magic” (see comments and updates at foot of page of this very insightful article by Jan Ploski; although written earlier this year, it is still right on in its honest attempt to share actual dev workflow usage, which is very refreshing). Behind that wall of magic, you know something is happening, but you don’t know what it is or even if you have it, while a team of rocket scientists need to be spending some quality features refactoring time, to come up with a semblance of self-contained re-usable and software configuration (version control and build) maintainable artifact sets packaged in modules and capable of being invoked by an installation profile.

This is true with or without the kit guidelines (the “library”?) and the monolithic “Build kit“. Both support a very specific set of use cases, and set up rigid rules for what should and should not be considered part of the library kit, something which will actually vary enormously according to the app architecture itself. And the Build kit starts out by hacking Drupal 7 core before it’s even been released and therefore really locks you in.

None of this can emerge as the fruit of an agile iterative and incremental team workflow, nor can it be popped back into a straightforward workflow as the app (and the install profile) evolves over time.

A very real problem, really

So, now that I wish to publish Project Flow & Tracker as a distribution profile right on http://drupal.org I find myself in a quandary which just should not be happening. Although there exist installation guidelines and several worthy examples to learn from (Innovation news, and the features based VoiceBox, Spaces 3 Demo, etc.) and even though I can install and enable contexts and features packaging modules from the install profile, the features I am invoking do not compartmentalize functionality in any reusable or maintainable manner; and Contexts and Features, which are great for many purposes and a godsend and a joy, do not provide any solution for the famous Drupal developoment workflow problem after all.

Conclusions and ToDo’s

So what conclusions can I draw in order to publish an installation profile for Project Flow & Tracker as something honestly capable of emerging as the result of an agile process, as part of real everyday work which follows industrial app building standards so that it is built as much as possible out of reusable elements, and may be customized in a modular fashion and subject to ongoing maintenance?

My conclusions are that not only do I need to separate UI features from “core library” or business object features, I actually need to strip features right out of the app entirely and use existing exportable tools (Views Bulk export, Ctools Bulk export, Strongarm, etc.), the only exception being where currently features are the only way to export app elements, or else make it much more straightforward. And even then they would be project artifacts capable of supporting multiple use cases, and not a single “feature”.

What are best practices in the software development industry anyway?

Best practices involve, along with the others mentioned throughout this article, bidirectional traceablity from business and use case model (captured requirements) all the way down to the chain of committed changesets, in order to be in a position to quantify socially useful work expressed as project artifact deliverables. In other words, given a user story, I should be able to visualize a log of committed changesets (definitely on the Project Flow & Tracker roadmap!). And I should be able to visualize dependencies using a build tool (I miss maven!).

Eureka: just export everything you can into project artifacts and use your version control system and build tool (and Project Flow & Tracker!) in order to achieve bidirectional traceability!

How can these best practices be applied to Drupal, specifically Project Flow & Tracker?

  1. Strip out all features from the project (easier said than done when in code and not in database).
  2. Rebuild functionality on the basis of modules built with exportables.
  3. Use Views bulk export and Ctools bulk export and CCK export and whatever you can find to get stuff into code artifact exportables, contained in one of the following:
    1. Installation profile.
    2. Module install script.
    3. Module update script.
    4. A minimalist feature if there’s no other way to do it, or if it is after all is said and done, the most straightforward way to do it.
  4. These modules must contain (in their .info files) explicit and non-mutually exclusive dependencies.
  • That’s why they cannot be features.
  1. Make sure these exportables based modules are as compatible as possible with the Install Profile API, and certainly invocable by an installation profile.
  2. Base ourselves basically on this workflow:

    http://nuvole.org/blog/2010/aug/24/features-based-development-workflow

    1. Recommended by Development Seed: “Nuvole’s Andrea Pescetti explains how to integrate Features into your workflow”
    2. But from slide 21 on, actually explains a much better alternative (also used by the dev folks at economist.com) which works whether you are using features or not:
      1. Start out with project.make and project.profile
      2. Use hook_update_N() for incremental development in order to put configuration in code
      3. This can be done with teams.
      4. As suggested elsewhere, refactor your build script (project.makefile) often!
  3. Enhance this approach with automated tools which can export configuration artifacts into code autmatically and have it captured by the version control system.
    1. Generate the drush make file for the project directly from the Drupal instance.
    2. Generate install profile

Will report back on how I publish the Project Flow & Tracker installation profile!

I will certainly let you all know what I did, and also, what I plan on doing in the future for a working stiff’s straight up web app dev workflow compatible with industry best practices.

Bibliography