Software process engineering, extreme programming and Drupal

I just would like to point out that it would be good for us all to differentiate between several key kinds of paths to be taken here, as a way of avoiding confusion and as a way of avoiding lost opportunities.

To become a Drupal "project master" (and the "must read" 12 IBM articles ( http://www-128.ibm.com/developerworks/ibm/osource/implement.html ) are a kind of "report from the other side", unfortunately not a pedagogical roadmap, rather a report of what it feels to have gotten over to the other side), there are several bodies of knowledge, areas of mastery, that must be attained:

Module mashup mastery

Theme mastery

Module reuse/creation mastery

Javascript and AJAX mastery

Forms mastery

Information Architecture and Software Development Process

… if not mastery, enough to avoid the prevailing high index of project failures…

Taking briefly as possible each one in turn:

Module mashup mastery

This is the kind of project practically everyone is talking about here in the group. Given a kind of site (ecommerce, educational, community networking), what are the permutations and combinations of modules and themes and theming which need to be invoked to acquire the necessary critical mass of functionality needed for the site. This is probably an important first start, and we will spend a lot of time here, and should (in order to create "best practices") but what I would like to point out that the effort of the group should not be exhausted in this one area, now dominating practically all comments.
But even here, we should evolve best practices, or at least a few schools of thought:

• The "don't be silly, do it with CCK and Views" school of thought
• The "don't be silly, do it with this shiny new module" school of thought
• The "use this tried and proven mix of modules" school(s) of thought

Theme mastery

This is not necessarily design graphics. It is an area usually postponed by the Drupal community, as mentioned many times. Even the fact that you can postpone it speaks extremely highly of Drupal: the famed "MVC" approach ( model (data) view (presentation layer) controller (logic and business objects) separates the presentation layer pretty cleanly from the others, and allows you to follow the best practice of developing the functionality first, and then the appearance.
If you read only one of the IBM articles, it should be "Getting started with Drupal" http://www-128.ibm.com/developerworks/ibm/library/i-osource5/ , which explains, in its incredible "Node-building sequence" diagram, but also in detail, how a view operation renders a page. Here you can see many things, and important for theme mastery is that the "theme" actually does two things for the presentation layer: with PHP it refines the model into div's (which data goes into which individual areas) and it provides styles (CSS) for refining the layout and appearance of those areas. Every Drupal Master (or at least one guy in the group) has got to be guru in this area.

Module reuse/extension/creation mastery

Here is where the group seems to be getting the short end of the stick; but this is natural, since most of us are beginners. But there should be sample projects relatively early on for us to know

• How to write a module
• How to refactor "theme" code to the module (i.e. affects all themes) level
• Erm… where do I put the code?

Javascript and AJAX mastery

jQuery, maybe script.aculo.us (what happened to that module???)

Forms mastery

Forms api, fQuery… wow, a lot to learn

Information Architecture and Software Development Process … if not mastery, enough to avoid the prevailing high index of project failures

JasonMR has already posed the question of Software Development Process, in his post http://groups.drupal.org/node/2195#comment-5836 , where he stresses "Planning" and "Gathering Requirements" as key topics.
I couldn't agree more.
But I feel that an archetypal Drupal Master (and there are probably very few indeed at the current time) should be proficient, that is, swim like a fish, in the following waters (and by extension, Drupal itself should be the first Drupal Master and have modules to wizard us through these areas):

• Business Modeling

In what business processes is the proposed web application to be inserted? What is the business/organization/group/individual user doing now to get the job done? Is using the project web application going to be much better? What does the web app have to interface to? Does data need to travel much from the outside in, or from the web app out?

• Agile web application approaches

Planning yes, but not top heavy planning. What are the best lessons about highly iterative and heavily prototyped processes where the code that is running, the system being built is the center of attention and not the planning; and yet there is planning enough to get the job done without getting lost (or knowing when you have finished)… get the global picture, write a few user stories, persona paragraphs, storyboards, site maps, use cases, wireframes… then list them in order of priority (the ones that get your ducks in a row as quickly as possible go first: risk mitigation)… then do them one at a time, writing an acceptance test before you start.
In a nutshell: avoid working without planning; and avoid planning without working.

• Test driven environment

There should be unit tests (each module, inc, script, each piece of code should have a "driver" that tests it; the devel module and macro could be sufficient for this) if possible automated. Since PHP is a compiled language, there is no "build" process per se, but there is a build process in the release generation in the project module… can unit testing and regression testing (automated testing it again after you have refactored code to make sure you haven't introduced new bugs) and acceptance test scripts be included in that module, to be run as an automatic part of the generation? Or as a part of the commiting process?
The acceptance tests, ideally written before the coding (configuring) starts by the the end user(s) and the technical people, are particularly important in order to answer the following infrequently answered questions:

• Are we building what the users really need and have requested? Are we implementing the requirements?

Does the site that is running have anything to do with what the users asked for?

• Are we done yet?

Drag the creative genius away from the keyboard.

• Change and configuration management

Here we are talking about two main fields, namely, version control, on the one hand, and software to manage changing requirements, anomalies, feature requests, etc., that is, change, on the other.
Again, can the project and project issue tracking module be tweaked for this? Already we have "pretty good" tracking; but the interface with the version control system is a) only concerned with getting the release out; and b) woefully inadequate; and c) not very extensible to non-CVS forms of version control, such as SVN and, increasingly, git ( http://git.or.cz/ ) (git would be great for drupal, but I would settle for SVN as a first step)!

• Environment

How to set up development, test, production sites, what tools (editor, debugger, build and packaging tools, etc.) to use; how to have skeleton Drupal installation (profiles?) (s) at the ready; how to avoid dreary and error prone repetition in basic configurations of new sites/projects; how to be able to come up (build) different versions of projects for different projects/customers.

• Usability

Don't make me think! http://www.sensible.com/chapter.html (read the sample chapter then explore the whole site). Definitely worthy of a class!!!

• Train the trainer

Docs, on-line help, that plethora of boxes all over Drupal admin labelled "… guidelines".
Training people in administering their own sites so they don't wake you up at night.
The first Dojo Lesson promises to document itself, this is a great step forward.

Anyway, we should have a "process" thread here in the group in order to make sure the global view of things and the overall objectives don't get lost in the shuffle.

Victor Kane
https://awebfactory.com.ar