Report back on a set of key DrupalCon Denver 2012 presentations

For many reasons DrupalCon Denver 2012 was a source of “fresh air” and excitement in the community, with emerging developments promising the “democratizing of technology” and a new head start in Drupal 7 site building (see What's new in panels and several references to the panopoly distribution below). All this of course in the context of the push towards Drupal 8 (currently estimated to be usable in approximately 18 – 24 months from now) and the debate as to what it will be like.

This (longish but I hope useful) article is an objective as possible review of the presentations I recognized as being absolutely key, out of the handful I was able to attend directly. I have organized them into a couple of main threads, as a prelude to a “What's up with Drupal” series of articles to be published in the near future covering my own recommendations and perspectives for site building and web application development with Drupal over the next couple of years.

The two threads covered here are “State of Drupal” (the official position on the future of Drupal and Drupal 8); and “Drupal 7 now for the rest of us” (promising developments, tools and resources for current work). The presentations I am reporting on, which I feel are key for the future of Drupal and the Drupal community, are the following: State of Drupal (Dries Buytaert Keynote), The Initiatives Formerly Known as WSCCI, Drupal 8 meets Symfony2, Directions for Drupal core Node.js Javascript and the Future, What's new in the Panels Universe, Open Academy: a higher education drupal product for departmental websites, and finally Delivering Drupal (Sam Boyer on devops and deployment).

Posted on: Last updated on: Written by:

For many reasons DrupalCon Denver 2012 was a source of “fresh air” and excitement in the community, with emerging developments promising the “democratizing of technology” and a new head start in Drupal 7 site building (see What’s new in panels and several references to the panopoly distribution below). All this of course in the context of the push towards Drupal 8 (currently estimated to be usable in approximately 18 – 24 months from now) and the debate as to what it will be like.

This (longish but I hope useful) article is an objective as possible review of the presentations I recognized as being absolutely key, out of the handful I was able to attend directly. I have organized them into a couple of main threads, as a prelude to a “What’s up with Drupal” series of articles to be published in the near future covering my own recommendations and perspectives for site building and web application development with Drupal over the next couple of years.

The two threads covered here are “State of Drupal” (the official position on the future of Drupal and Drupal 8); and “Drupal 7 now for the rest of us” (promising developments, tools and resources for current work). The presentations I am reporting on, which I feel are key for the future of Drupal and the Drupal community, are the following: State of Drupal (Dries Buytaert Keynote), The Initiatives Formerly Known as WSCCI, Drupal 8 meets Symfony2, Directions for Drupal core Node.js Javascript and the Future, What’s new in the Panels Universe, Open Academy: a higher education drupal product for departmental websites, and finally Delivering Drupal (Sam Boyer on devops and deployment).

In all cases links to the presentation page are provided, where videos and in some cases slides are available (recommended).

Keynote – Dries Buytaert

Presentation: http://denver2012.drupal.org/keynote/dries-buytaert

Slides: http://buytaert.net/files/state-of-drupal-march-2012.pdf

The speaker expressed his concern about how Drupal might win the hearts and minds of people as a product, given that fashions change over time. Some products, like Kodak film and the Palm Pilot, directly became obsolete. Those that survive the sands of time are those that are based on continuous innovation. Google, Apple, Twitter and Facebook, and even GM, to name a few, continue to be innovators. Now, Drupal 7 is approaching maturity, Drupal 6 is in decline, while Drupal 5 is definitely obsolete. While no mention was made of Drupal as a whole, the concept of “innovation extensions” was presented as a way of breathing new life into existing products. Some companies bring out new products, others fall by the wayside as their products become obsolete.

Just as Apple brings out new products, Drupal must innovate in its features set. Drupal self-reflection is necessary to decide which features need to be launched. A video was presented which despite its overwhelmingly positive attitude towards Drupal did manage to list some drawbacks: its rudimentary authoring experience, especially for those using the CMS on a 9-5 basis; its aging web development framework (in spite of course of being the best CMS technically speaking), and the lack of Drupal talent (in spite of some efforts to remedy this). Strengths: the community, being flexible and based on open source, can collaborate and innovate together. Opportunities arise from the growing share of CMS based sites and the growing share of Drupal in that sector; and from the huge growth in Mobile and Drupal’s participation on mobile platforms, which will experience a twenty-five fold growth over the next five years. At the same time, Drupal faces the threat of being passed by as time passes.

So “it’s time to kick ass with Drupal 8”, with new features aimed at three kinds of users: Authors, Site visitors and Developers.

For developers: a modern framework based on a new and increasing dependency upon Symfony, in a process for the back-end parrallel with the adoption of jQuery for the frontend. This will attract more developers to the community.

For site visitors: a great mobile experience. This will depend upon responsive design in an HTML 5 context, for browser based web apps, and on the use of RESTful web services and SDKs to support native mobile apps.

For content authors: great authoring tools. Right now Drupal leads technically but lags woefully in the authoring experience it offers. This will be rectified by putting “more in core” (in-line editing, improved content administration tools, drag and drop, better media support, page and layout building tools; and by promoting the availability of Drupal distributions offering specific solutions.

Drupal 8’s project plan slates a December 1, 2012 feature freeze, a Frebruary 1, 2013 code freeze and an August 1, 2013 release (18 months from now).

Each of the six initiatives are in varying states of progress in their analysis, design and development phases.

It’s time to kick ass with Drupal 8 to win the hearts and minds of users, authors and developers.

The Initiatives Formerly Known as WSCCI

Presentation: – http://denver2012.drupal.org/content/initiatives-formerly-known-wscci

[Edit: See WSCCI Routing in Denver for groundbreaking news on how Drupal 8 *will* be using the Symfony2 Routing component; thanks to Larry Garfield for his correction on this]

“The Web Services and Context Core Initiative (WSCCI) has been split in twain! The new plan is two initiatives, one focusing on the Symfony2 kernel and web services and the other focusing on blocks and layout. Larry Garfield and Kris Vanderwater will be speaking on how the new initiatives will play out, what they mean for core development, and the next steps that people can take to help.”

The kernel will support “blocks everywhere” and panels in core, a pull based layout, esi-style page building. The context (i.e. node id, user id, etc.) will be sent along to the controller as a parameter, so each block will have context.

[Victor Kane: see http://blog.redfin.com/devblog/2010/05/esi_and_caching_trickery_in_varnish.html for background on ESI tags].

This will be based on the Symfony (all references are to Symfony2) library for HTTP handling (HttpFoundation, HttpKernel, and Routing – see next presentation Drupal 8 Meets Symfony2). The routing will privilege a REST-first, Mobile first approach. Common service-type requests will also be privileged and the page will be only one among severy kinds of requests.

The Layout initiative will involve a complete reworking and redefinition of blocks system and provide a plugin system supporting contextual plugins. A complete layout system will be built and existing out-of-the-box themes will be transitioned to it.

Drupal 8 Meets Symfony2

[Victor Kane: This presentation was important, in that it was given by Fabien Potencier (fabpot: Sensio CEO (sensiolabs.com) and Symfony project founder (symfony.com)) and Lukas Kahwe Smith (PHP 5.3 Release Manager and Symfony2 core contributor), so it’s straight from the horse’s mouth, so to speak; and in that it sheds some light on which parts of Symfony will be incorporated into Drupal 8 and which later, or at least on the priorities which have been discussed.]

Presentation: – http://denver2012.drupal.org/program/sessions/drupal-8-meets-symfony2

The slides supply great detail, including availability of code and code samples of each component, and are available here: http://speakerdeck.com/u/fabpot/p/symfony2-meets-drupal-8

For the speakers, Drupal has a steep learning curve for its use in development and to be productive in Drupal core, and the effort applied in climbing that curve would only be useful for Drupal. On the other hand, others build on top of long-established OO design patterns, and leverage PHP enhancements. The following quotation is alluded to in the slides:

“The complexity of the custom code that’s used and the non-standard architecture combines to create a barrier to entry for developers new to Drupal (both experienced and novice developers alike).”

“While these changes may seem large, and some of them are, we believe that they will achieve the right balance between empowering Drupal’s design and architecture, and moving toward more modern, standard, well-tested code and techniques to empower a new generation of developers. I hope you are as excited as we are!”

from http://buytaert.net/the-future-is-a-restful-drupal)

Coincidence with Dries’ Keynote is expressed: Symfony could facilitate refactoring towards a “framework” core, allow developers to focus on the CMS itself, allow for easier integration with other frameworks, and attract more developers to the Drupal community.

“Symfony” of course, is Symfony2.

http://symfony.com/

http://symfony.com/components

“Symfony is a reusable set of standalone, decoupled, and cohesive PHP components that solve common web development problems…. an object oriented set of classes… compatible with PHP 5.3 and later… a full stack web framework”

Many developer tools are using Symfony: Behat, Doctrine, Propel, PHPUnit, Jackalope. Frameworks include Silex and PPI 2. And Products include easybook, Midgard CMS, Zikula, phpBB, and… Drupal 8.

Components:

  • ClassLoader

    • PSR0

    • classes must follow conventions to be loaded by this

  • HttpFoundation

    • Replaces PHP native global variables and functions

    • Provides encapsulation of request and response methods

    • Provides object-oriented layer for messages in session management

  • Routing

    • [Edit: See WSCCI Routing in Denver for groundbreaking news on how Drupal 8 *will* be using the Symfony2 Routing component; thanks to Larry Garfield for his correction on this]

    • Not going to be used since Drupal “has to have backward compatibility and it has too many routes”

  • EventDispatcher

    • Will not be replacing the Drupal hook system until after Drupal 8

  • HttpKernel

Handles the request/response workflow logic. Listeners can respond to request/response events so that the appropriate controllers are invoked, and the response is generated. In the case of an exception, listeners convert the exception to a response. Quite detailed treatment available in slides/video.

The HttpKernel makes Drupal interoperable with any other software using the same.

There are testing and Caching components.

Edge side includes (ESI) support.

  • DependencyInjection

  • BrowserKit

  • CssSelector

  • DomCrawler

Rather than simpletest, phpunit might be more adequate as a testing framework for Drupal 8.

Directions for Drupal core

[This presentation by Dries is key because it brings to the fore the debate over big core, small core and the compromise lieutenant (packaged) core, a debate which must be closed to allow Drupal 8 develoopment to move forward clearly. Apart from the main points of the presentation, the video and the slides are extremely important also to appreciate the questions asked and answered which formed the brunt of the proposal.]

Presentation: http://denver2012.drupal.org/content/directions-drupal-core

Slides: (link found on http://buytaert.net/state-of-drupal-presentation-march-2012 )

http://buytaert.net/files/state-of-drupal-march-2012.pdf

Statement of the three positions:

Big core

Most users want a big core. [??? Is that really true: they want packaged apps so they can start using it right away, but that can be accomplished through any of the core architectures presented?]

Core tends to grow as more functionality is added, and Drupal core got much bigger going from Drupal 6 (i18n, update status and ahah), to Drupal 7 (CCK, Imagecache, Admin role, UX, Vertical tabs), and on to Drupal 8 (Views, Module filter, WYSIWYG, HTML5, Mobile, Drafts, Pathauto, Flag, Date, Rules, Media, Inline editing, Panels).

That is the big core position [that’s not big core, big core is the product]

Pros:

  • Much better out-of-the-box capabilities, comparable with competition

  • Only one team, repository and issue tracker to coordinate

Cons:

  • Complexity of core grows

  • Current core developers don’t want to be responsible for features they don’t care about

The core developers are used to working on their subsystem and new subsystems are not what they are always passionate about. We would need to attract core devs

Small core

A lot of distribution authors and some core developers want “small core”.

In Drupal 6 you had to do a lot of work to override core behavior, but in Drupal 7 this has improved a lot, and D8 with Symfony and the other initiatives it should be easier for people to undo things.

Drupal Core would be a framework exposing APIs, and the different distributions would be built on top.

Pros:

  • Much better product/framework separation

  • Only one team, repository and issue tracker to coordinate

  • That team responsible for far less

Cons:

  • Lots of wheel re-inventing between distributions, lack of collaboration

  • Users faced with multiple choices when they don’t even know what Drupal is yet

  • Drupal can’t complete with “pure” frameworks

  • Symfony doesn’t power 2% of the web

The reason Drupal wins today is because of its unique combination of framework and product. A framework has the disadvantage of not being able to penetrate a large percentage of the web except in bespoken websites [not if it gets in the back door of Drupal and others].

“Lieutenant Core”

What some core develoopers want: a small core with additional selectable packages.

  • Drupal

    • Views

    • Panels

    • Media

    • Test

pros:

  • Much better out of the box

  • Drupal N=1 ships with key contrib modules ready to go

  • no getting blocked on core commiters/core review process

Dries’ current vision around Drupal core

Dries vision of drupal core architecture for Drupal 8

“The core of Drupal should have all of the infrastructure that people expect in a website. And that goes from DX (developer experience) … that means adding those things to core, the bigger core.”

The things people need for building websites. Things like pathauto which everybody uses, things like date.

And content editors (UX, WYSIWYG, Layouts, Inline editing).

And site builders: (Simple) views, pathauto, date.

Some people still hate wysiwyg and js and maybe it’s time to let go.

Phoenix

Presenting the Phoenix distribution, the Drupal 7 “pressflow of UX”

http://drupal.org/project/phoenix

“My goal is to maintain a distribution called Phoenix and pull in some of the best authoring tools. …a Pressflow for UX. Kind of build a prototype of what Drupal 8 might look like in terms of authoring experience. Let’s build it for seven and move it into Drupal 8 core”.

Node.js, Javascript and the Future

Presentation: – http://denver2012.drupal.org/program/sessions/nodejs-javascript-and-future

Slides: http://www.slideshare.net/miccolis/nodejs-javascript-and-the-future

The presentation was given by Jeff Miccolis http://denver2012.drupal.org/users/jmiccolis http://drupal.org/user/31731 of Development Seed http://drupal.org/profile/profile_companies/Development%20Seed

Development Seed, having been a small Drupal shop in Washington D.C., tended to do a lot of NGO, Data and Collaboration sites with an international bent (multilingual). They created Open Atrium, a group collaboration suite which they still use internally. It was based on Features, Context and Strongarm modules [Victor Kane: huge contributions to the community].

Development Seed at one point, focusing on Open Data, Visualizations and Maps, did a “pivot” as a company, to http://mapbox.com/ . Foursquare recently switched to their maps. (see Designing Fast and Beautiful Maps http://denver2012.drupal.org/program/sessions/designing-fast-and-beautiful-maps session)

Rather than getting into the real nitty gritty of Node.js (covered in many community videos) the focus here is the telling of three “war” stories on things that are hard to do in PHP and Drupal and would have been easier in Node.js; and two lessons learned from moving from one big system (Drupal) to the next (Node.js) as eco-systems.

Node.js http://nodejs.org/ is a server platform built on V8, Chrome’s JavaScript runtime http://code.google.com/p/v8/

Story 1: Sending lots of emails

Open Atrium had to send lots of mail as it is a collaboration suite, with checkboxes for all group members to notify of posts via mail (like basecamp); Open Atrium sends single emails, digest emails, sms, xmpp (jabber) twitter, etc. and channel independent messages delivered by mail, sms, jabber, twitter, etc. In drupal, you hit save, and naively send the email right away in the submit handler. But it takes time to send the email:

  • sending a single email takes 50 ms

  • 28 users – 1.4 seconds

  • 600 users – 30 seconds

This can be done by cron, but there is a lack of error handling. In node.js the function is passed to the email address object, each individual mail sent has its own non-blocking error handling, and the whole process is speeded up ten-fold: 600 messages with node.js takes 3 seconds. The presenter compared php to node.js code and organized an active role playing exercise to demonstrate how the asynchronous non-blocking process does not have to wait for i/o. “Stop waiting around”

Story 2: Handling feeds

Development Seed also developed a news aggregation system called Managing News, which aggregated large numbers of geo located stories. So the task was:

  • fetch rss feeds

  • fetch original article

  • tag items (calais)

  • tag items (watchlist)

  • tag items (wikipedia)

  • geocode items

with thousands of feeds, millions of items, gigs and gigs of data (testing environment had a database of 4.5 GB). The task was handled by cron, but there was not even enough time in a day to guarantee all feeds would be hit. So, 4 retriever dogs got batches of 50 items in a python based multi-threaded system and perform the task. The times were still slow, on average:

  • retrieve original story: 300ms

  • tag: 100ms

  • geocode: 150ms

  • total: 550ms

That’s half a second, and during that time the server is idle, not doing a darn thing.

Solution: instead of four run a whole pack of dogs. That didn’t work, or was not workable at the time. In retrospect, and based on experience working with Node.js, the retrievers should have been replaced with a single hyperactive squid (see search engine on this metaphor for the event loop and how Node.js works on your server and check out the first presentation: http://www.slideshare.net/simon/evented-io-based-web-servers-explained-using-bunnies ). The gist is that the asynchronous event loop, as a single process:

  • The squid runs up and down as fast as it can dealing with each item in turn.

  • It fires off any long running I/O operations and then moves on to the next item.

  • When the I/O operation reports progress, it does a little more work on behalf of the corresponding item.

Anything that leaves v8 takes a callback: “I’ll take my time doing it and then call you back with the callback when I’m done.” The kind of stuff that leaves V8: filesystem, network, stdio, timers, child processes … 100% async. See code samples from slides. “This is what asynchronous code looks like”: You get to ask your machine to do more work.

The limiting factor is now the cpu; the classic limit in php is memory; with node.js it is also cpu.

Story 3: Big files, long sessions

Files of several gigabytes, with uploads (http sessions) lasting hours. In php you tweak upload_max_filesize, post_max_size, max_input_time, max_execution_time and are able to manage, say 500 MB, but you open the door to DOS, and even with app bloat you are never going to be able to upload files measured in the gigabytes. “If only we could stream”. Streaming is like a bucket brigade, but with php all the buckets arrive at once instead of being passed individually (stream) where each bucket doesn’t use that much memory, can be written to disk as it comes in or sent off to s3. Node.js code example shown for this (note that Node.js does not run embedded in a server, but rather you instantiate a server in the Node.js code itself in just a few lines). See https://github.com/felixge/node-formidable (“A node.js module for parsing form data, especially file uploads”), developed for http://transloadit.com/ a file uploading and encoding service based on Node.js

CouchDB http://couchdb.apache.org/ mentioned as a form of No-SQL schemaless document store that can be used to persist streaming data; and the fact that it has a __changes() function that feeds you new data over http when it has some when you are subscribed to it. That’s how MapBox implements map uploads, directly to s3, with a record saved to CouchDB referencing the new file, which is then propagated to long (running for days and days on-end) Node.js processes which announce new maps ready for download.

So non-blocking I/O over a single process event loop allows you to do amazing things.

Story 4: Package management

drush and drush make for Drupal allows for great package management. Now, on d.o., each project requires its own namespace, and an inclusive project policy authorizing folks to manage a project on d.o. (drush make handles the packaging), so all projects are in one spot so people can interact and contribute to projects, with naming issues resolved, makes drush make a way to stay sane. But drush is a relatively recent development, it would have been nice if there had been a php based project capable of solving these problems for us, like pear. But it still has a high threshold for new projects. Now, what if pear were wildly inclusive (i.e. easy to get an account), awesomely useful and successful? Well, in the Node.js world, there is npm the node package manager, which is all of these things. It is only two years old and already has double the number of projects than d.o. The equivalent to a drush make file is package.json present in the root folder of each project.

Story 5: Nice hammer

Use the right tool for the job (drupal vs node.js). Node.js bad for: computationally heavy tasks (hogs all cpu of system) and database implementations. Node.js: good for interacting with other services, like databases, mail servers, web services, web clients (socket.io)

A couple of resources on why node js is cool:

What’s new in the Panels Universe

Presentation: – http://denver2012.drupal.org/program/sessions/whats-new-panels-universe

Slides: http://denver2012.drupal.org/sites/default/files/What%27s%20New%20in%20the%20Panels%20Universe.ppt

[Victor Kane: With panels in core for Drupal 8, and the exciting need to have a clear site building path for Drupal 7, and coverage of the exciting Panopoly distribution, this session announcing recently released Panels awesomeness for Drupal 7 proved to be one of the most important sessions, and drew an attendance rivalling some DrupalCons in the not so distant past.]

The Panels Universe:

  • Panels (duh)

  • Page Manager (CTools)

  • Panels Everywhere

  • Panelizer

  • Fieldable Panel Panes

  • Field API Pane Editor (FAPE)

  • Entity Revision Scheduling (sorta)

  • Panopoly

Panels (duh)

Within the Panels suite of software, panels itself is now just the layout manager, it does styles and drag and drop stuff and provides a central way so that the panels system may be re-used in other applications. Its major component is Page Manager, now split out.

New in panels:

  • Redesigned IPE (Chapter III)

    • User experience, it looks much nicer now

    • Can change layout

    • Can change pane styles

  • Pane locking

    • Prevents content people from moving structural layout panes so they don’t have to worry about breaking the site while editing content

    • Can lock panes to a region or make them immobile

Page Manager (CTools)

A GUI for hook_menu. Its input is the URL and other info hook menu normally receives and it mainly outputs the rendering of the content for that URL. The primary output vehicle is panels but you can also specify http response, EclipseCG’s context admin module http://drupal.org/project/context_admin, 301 redirects, 403 access denied, 404 pages, pretend the path doesn’t exist…

Panels Everywhere

A replacement for the block module entirely and for the page template. It needs a lot of work, I am hoping to redo the UI in the next year to make it more friendly especially since there is an initiative to get a lot of this layout stuff into Drupal core. My actual hope is that the panels module will cease to exist but that the rest of these will continue to exist so the core can do the layout stuff and contrib can use that layout stuff in ways that core might not be able to do. That would put the core functionality in core and expand on that in a number of ways.

Panelizer

The new stuff has mostly been funded by clients, Panelizer and other modules in order to provide a good experience for content editors. Panelizer is panel nodes on steroids and with context. Also part of the Chapter III funding (merlinofchaos worked about 200 hours in three weeks on this).

  • Like Panel nodes, but with context

  • For any node or user or taxonomy term

  • By bundle

  • With defaults

  • That can be picked by the node editor

We allow you to have a panel layout for any node and it will provide a default layout or multiple default layouts and you can choose which one to use. Every aspect of the system has permissions so I can say only executive people can control the layout, so these nodes will get the default layout and nothing else or I can give them permissions to only select one of the layouts and nothing else or I can give them full permissions to modify everything but the layout and the context per node. And each node bundle gets its own set of permissions. So you can be very very granular with permissions or remove them and let people do whatever they want depending on the needs of your content people and how much true separation you have. We made it entity generic so it comes with plugins to work on users or taxonomy terms and outside of the node the plugins are really short, 50-100 lines of code (how to export, how to save). That means that if you’ve got custom entities on your site, say product entities from commerce it isn’t too hard to extend this system to non-node entities and then use that to penelize anything you want. Very useful for commerce where you can have very complicated displays as part of catalogs.

I think Panelizer is a revolution in the paradigm because it presents users, the content people, with a much slimmed down interface. One of the biggest complaints about panels is that you can’t actually give it to your content people because they end up in the Page Manager UI which is very complicated. So your content people will never have to see the word “Variant”. And panelizer covers some 80% of the use cases for variants which means you might not use variants yourself even though they will be used behind the scenes. And that strips away a level of complication that many people are afraid of.

Fieldable Panel Panes

Entities that can be used in panels directly, analogous to the Bean module http://drupal.org/project/bean for blocks.

  • An entity that can be added as a pane

  • Create on the fly or in administrative UI

  • Full access control

  • Supports bundles through a hook (no UI)

  • Like BEAN module for blocks

In Drupal 7 one of the key things about an entity (node, user, taxonomy,…) is that you can add fields to an entity and because we have is (still a long ways to go but) massively improved file handling. One of the things that is really hard to do is add an image inside a pane, since it has to be stored. You can now set up an entity with whatever fields you want and your content people can create that entity either from a custom UI or directly inside the panel. You can add bundles to this (media, image). You can say it is reusable, so it will show up in content like that (you can say in what category it will show up in). Or you can say that it is not reusable and it will only be in that panel.

Access control can control who will be able to view that pane as well as who will be able to edit that pane. In Open Academy we had people in different departments, like Health and News, they wanted to create panes that were usable only for those departments. All have permission to create fieldable panel panes, once one is created, only people in the same department can edit that particular pane, it’s their department’s pane and no-one else will be able to mess with it. That’s just by using the selection rules that are everywhere in panels and pretty easy to figure out.

Field API Pane Editor (FAPE)

A really tiny module that turns out to be really nice:

  • Form to edit a single field on a single entity

  • Contextual link on a panel to hit that form in the overlay.

  • That’s it!

In panels if you are looking at a node or an entity that has fields, there’s a pane which just displays a single field, you can select the format and control how it looks. This adds a contextual link to that to bring up in the overlay module a page that edits just that field in that entity. And offers you the ability to handle revisions. When you have complicated nodes, when you’re a media company your node can have lots of little things it’s really convenient to be able to edit things isolated from each other so that you’re not overwhelming your content people. They get to look at the content and say “I want to edit just this little piece” (applause!). In the future I want to be able to create bundles of this so can put more than just one field in your boxes, you can create little groups of fields that have panes. But right now it’s one field, but still that’s really handy.

So FAPE is a tiny module and the back-end (the actual editing in the overlay) isn’t tied to panels at all so if you don’t care about the panels aspect you can still create your own interface to that edit form, it handles proper access control so you can use it in your own system without panels.

Entity Revision Scheduling (sorta)

http://drupal.org/project/ers

Not really panels, but is being used heavily with panels.

  • Create a ‘draft’ revision that is not the normal revision used.

  • Schedule when a revision is set to the current revision.

  • Works for Nodes as well as Fieldable Panel Panes

  • Still need a couple more features and a big valuable patch by recidive to unpublish needs review!

    • Breaking news: it’s in the 4/2 new beta version ers-7.x-1.0-beta !

iVillage funded this module because they had a need to publish content at scheduled times (“I want to publish this at midnight”). While two or three modules do this, like http://drupal.org/project/workbench_moderation and others, and they all have problems. Entity Revision Scheduler has problems too, but a particular set of problems to match the requirements: First of all it is entity agnostic; much like Panelizer you can attach this to whatever entity you want. They needed this for the Fieldable Panel Panes, something Workbench moderation simply would not have been able to do. But more importantly I took a different approach. Their approach is triggered when the node is viewed (content swap out), my approach swaps out content when you go to edit. So the published one is always the current revision, but the one that is being edited is another revision. “Sneaky code” in the background makes sure that when you save that node the “right things” happen. The nice thing is you are editing the revision and it is completely invisible to the user.

Video: The best way to describe what this module does is to watch this 2 minute video (no sound): http://yfrog.com/ng44160214z

Scheduling works on cron and on view, so it will actually check so your scheduling happens immediately as needed.

Since it works with entities this could be extended to work on Commerce products if needed to revision your catalog and make massive changes that go out at a certain time.

Panopoly

http://drupal.org/project/panopoly panopoly 7.x-1.0-beta2 last updated March 21, during DrupalCon to include starter content!

A Drupal distribution funded by work being carried out by Pantheon and Chapter III: Matt Cheney has been working overtime on Open Academy, their distribution for education, and they realized that if you just took out a few things it could be a pretty generic distribution and I have been talking about a Panels based distribution for years but the technology hasn’t been there and the UI hasn’t been there so it has never materialized but as of about a week ago this has materialized in beta form, it’s not quite ready yet but it is on drupal.org and you can install it and it’s also on Pantheon and you can spin up a Pantheon site. A lot more will be changing over the next few weeks but it’s there now.

  • Panels-based (but more than just panels) site-builder distribution

  • Based upon OpenAcademy but generalized

  • Still in its infancy

  • Should provide a fantastic starting point for site building.

It comes with Panels and Panelizer, of course, but many other things also, for example a WYSIWYG editor and a bunch of node types and a lot of stuff they find themselves using over and over again when they are building sites. Will really ramp things up for people and make getting started a lot easier.

I’m excited about this and I’m hoping that over the next few months this will grow into something a lot of people use.

These modules all work well with Deploy module to get around the limitation on exporting entities in Drupal 7, so there is a clearly defined way to move content from staging to live sites

See demos and questions (video):

New better designed IPE interface to change layout, etc., easier to assign panes to new regions when you change layout; the IPE properly locks for changes, so another user will see popup announcing blocked state (choice of blowing away lock with AJAX reset). (Do not confuse this with locking a pane in a region, which functions in the backend along with the permissions configuration).

If you move panes, panes locked to a region will move as a unit. Fine granularity over all these permissions.

Locking from deletion will show up in a couple of weeks as a new feature.

CSS specified in panels is placed in the db but also in the filesystem (./files/…), and is aggregated with the pages css cache as part of the normal Drupal CSS cache process (almost the same as aggregation works).

Panelizer configuration in admin (content authoring).

Panelizer demo at minute 33

Panels choice means you can create a list of layouts that can be applied to a node type, etc. You can choose the layout when you create a node.

In the page manager view, layout and content is absent in panelizer variants, but you can still use rules and, say abort or provide fallbacks if panelizer is inappropriate; otherwise you would just use the panelizer UI.

By role panelizer defaults and user choices can come into play using the page manager backend interface (in spite of roles not being bundles), but may prove problematic if a user has multiple roles.

The ERS module works with Panelizer but not IPE..

Fieldable Panel Panes demo at minute 43

Notice Ctools content and these kinds of content are not nodes, so they are not available in Administration / Content. It could be, but VBO could be marshalled to manage these new kinds of content.

Open Academy: A Higher Education Drupal Product

Presentation: – http://denver2012.drupal.org/program/sessions/open-academy-higher-education-drupal-product-departmental-websites

“The university department is at the center of academic life. All across higher education, tens of thousands of departments have organized themselves to teach courses and perform research. As it turns out, most departments need a website to promote their work and each of those websites is basically the same. Open Academy is an academic departmental website in a box. Built by Pantheon, Chapter Three, and the University of California at Berkeley, we have baked in critical functionality around departmental news, faculty profiles, publications and presentations, events and calendaring, courses, resources and links, video, social media, and degrees and programs. Out of the box, Open Academy lets a department get an amazing and extendible website. As a bonus, Open Academy was built entirely with the Panels module utilizing the In-Place-Editor, pane level fields override, style plugins, layout aware pane displays, and much more. Come see a best practice Panels Drupal Product. This session will be given by Matt Cheney (from Pantheon and Chapter Three) and Brian Wood (from the University of California at Berkeley).”

http://drupal.org/project/openacademy

See http://drupal.org/node/207036 about increasing memory limit to 256 MB at least temporarily to avoide install error.

Install for free on Pantheon: https://www.getpantheon.com/

http://apps.chapterthree.com

http://chapterthree.com/open-academy

www.drupal.org/project/panopoly

Brian Wood (UC at Berkeley)

Used Aegir on SliceHost to host Drupal sites. Then moved Aegir back onto campus servers. Now that service is being retired due to financial questions. Outsourcing of hosting led them to migrate to Pantheon. 125 Berkeley sites, probably closer to 200 sites, 25% of all departments. Need for support on migrations and other maintenance.

Drupal Hosting needs: cross campus standardization, support for site maintenance and updates, performance and scalability issues.

Drupal Development needs: Overcome high cost and inconsistency of custom development, the difficulty of finding Drupal site building talent and the lack of standardization in development and the constant reinventing of the wheel.

Lack of consistency in backups, version control, centralized and mixed mode (CAS, Drupal based) authentication, too many homebrewed themes, no cross campus look and feel.

Solution: outsourcing to multiple service providers, together with best of breed best practices hosting, leverage the install profile and distributions, using them to reduce site development time. We want to make Drupal site building both easier and (campus, accessibility) standards compliant.

Impressed with high performance Drupal hosting and best practices dev/test/live workflow on the Pantheon platform, and were then exposed to Open Academy, with which they started to investigate via initial development using the distribution, also making available a lightly customized vanilla Drupal 7 starter set (“start state”) distribution as an alternative. The direction forward also includes centrally managing a UC Berkeley Drupal apps server to share code campus-wide.

pantheon.berkeley.edu – Easier SSL (easy certificates) and CAS which allows for centralized CAS wild card allocation of subdomain permissions; together with a UC Berkeley-specific Pantheon dashboard.

Matt Cheney (Open Academy)

Universities often have bad websites (departments, research groups, faculty members, events, etc.) in spite of having great content and content authors. Part of what Open Academy does is to make spinning off sites easy and configuration of these sites tasks which do not require developers. Faculty members or anyone not having specific Drupal skills should be able to spin off and configure their own site. The aim has been to “democratize technology” so that a department doesn’t have to spend 20-30 thousand dollars to have a whiz bang site, and you can have a powerful solution customized to your needs and with a cross-campus look and feel. Open Academy arises from Berkeley university site building needs seen in conjuntion with those of other universities, all with varying budgets, with the overriding concept that a Drupal product could meet these common requirements.

The Open Academy product seeks to privilege the content creator who can get their job done without going to the backend or having to be a technical whiz, with everything on the front end, with previews and easy to use.

In actual fact the philosophy is that everyone should be able, essentially, to make their own site for free without having to heavily customize. You can go download OpenAcademy, install it and run it and have a great site. Also, Drupal is not going to win 20% of the internet without solutions of this kind.

The basic architecture is Drupal 7, together with a selection of core and third-party modules, upon which is based the Panopoly distribution, which in turn forms the basis for the Open Academy distribution itself. Panopoly provides a standard base layer to enhance standard Drupal functionality, for example with a WYSIWYG editor, not so easy to set up and configure in standard Drupal, or faceted search (which downgrades to stock Drupal search if you don’t have Apache Solr installed), or admin dashboards (which extends http://drupal.org/project/total_control); on a foundation of Panels magic (Panels, IPE, Panelizer, Fieldable Panels Panes, Page Manage, Panels Breadcrumbs and Panel Layouts. It is downloadable right now, and that is what Open Academy uses to build all its stuff. I think it is the future of Drupal in layout and content management, where Drupal 8 is headed. I have been using Panels since 2007 and thought it was the future of Drupal and that is more true now, for products, than it was for building sites.

You get a whole slew of layouts without writing any code which are responsive to boot, you get the IPE (in place editor) for changing the site look and feel without having to go to the backend or write code, you get previews, you get the WYSIWYG editor and content creation experience more straightforward for people who aren’t used to Drupal. Online now at http://drupal.org/project/panopoly also available on Pantheon as an install.

Open Academy features:

  • Open Academy Core

  • People (directory of profiles)

  • Publications (integrates with biblio module http://drupal.org/project/biblio for citations)

  • Courses

  • Events

  • News

These half-dozen use cases are what universities build over and over again, and there are more (media, image gallery, FAQ, etc.) planned for the distribution. These are done as apps, which is an extension to features, allowing for installs and updates via an app server. Universities can add their own as well.

Open Academy comes with a starter wireframe theme built with a template approach allowing for theme customization. See http://www.chapterthree.com/blog/nica_lorber/design_drupal_template_approach “We came up with a list of elements inherent in 90% of Drupal sites, some basic, and some Drupal specific, to serve as a tool to make sure every last element had a style” (Nica Lorber’s article has the complete list, and cites the very usefule Zivtech style guide: http://www.zivtech.com/zivtech-style-guide). All elements are responsive.

Live demo at minute 31:00

Specific content edit experience demo at minute 36:30

Working groups supported via creation of Basic Page instances. The layout and content each working group (including panes based on views or various apps) can be previewed and modified thanks to Panalizer and Panopoly. Tremendously powerful technique for prototyping also for UX and designers (i.e. non-coders). These panes can also be reusable and together form a library of elements.

TinyMCE used (same as WordPress) with battle tested layout of buttons. Better formats module.

Pathauto config privileged.

Media module (2.x version of dev) consensus decision moving forward.

Menu block module used for submenus.

Thanks to Panopoly, the individual basic page layout can be customized without coding, etc. or any configuration on the backend.

Stylyzer not included but could be.

Spotlight carousel of images, based on the spotlight Drupal entity (analogous to a nodequeue).

Near term roadmap for integrating/importing existing data feeds: http://drupal.org/project/feeds

Delivering Drupal

Presentation – http://denver2012.drupal.org/program/sessions/delivering-drupal

[Victor Kane: In the absence of slides I have researched references and links to all the concepts and tools]

“Brutal truth: as a community, Drupal’s understanding is pathetically behind much of the world when it comes to devops and deployment. Most shops, and even many larger companies with dedicated Drupal teams, use one-off, unstructured, error-prone systems to get code from development to production… Solutions tend to come in two forms:

  • Better project tracking software.

  • Better human communication.

“At the end of the day, what matters is getting code from development to production – “delivering” it. Doing that properly means demystifying and solidifying the steps in that process, which entails tools like Git, Jenkins, Puppet/Chef, and Capistrano. Or, in a word: Devops. Building a system that can deliver code in a way that informs and includes all the relevant stakeholders, from developers to product owners, requires the right tools linked with the right mentality.”

Continuous Delivery

Continuous integration with a purpose. This is a major commitment and very hard to do.

See http://devops.com/2011/02/19/jez-humble-continuous-deliver/

DevOps

“At its heart, DevOps focuses on enabling rapid, reliable releases to users through stronger collaboration between everybody everyone involved in the solution delivery lifecycle. One of the important results of this collaboration is the application of agile practices to the work of operations, in particular aggressive automation of the build, deploy, test, and release processes for hardware and system environments.” – Jez Humble http://www.cioupdate.com/trends/article.php/3933106/Tired-of-Playing-Ping-Pong-with-Dev-QA-and-Ops.htm

Jez Humble is the author of the book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) http://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912/ref=sr_1_1?s=books&ie=UTF8&qid=1333458516&sr=1-1 and works for ThoughtWorks

See his blog http://continuousdelivery.com/ )

Automation of all the steps (build, deploy, test, release) from when your code gets committed to when it is delivered.

Any site cannot be reduced to the live instance, but rather is at any one time a constellation of dev, test, staging and live sites. The production site is the end point of a process that involves a lot of different environments in many stages and instances. How does code get from one place to another in this process? Magic? DevOps!

So “dev complete” is not the same thing as “live”. Code between these two stages is still in someone’s mind rather than delivered.

Deployment Pipelines

  • Captures each stage code goes through from dev to prod

  • Locks those steps into a structured, semi-automated process

See http://continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/ and particularly http://continuousdelivery.com/wp-content/uploads/2010/09/optimized_pipeline.png showing automatic and manual approval pipelines for code.

For any given project we articulate the deployment stages and we move code through them in a structured way.

There is a Jenkins http://jenkins-ci.org/ plugin for Continuous Delivery. References:

https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin and particularly https://wiki.jenkins-ci.org/download/attachments/54723106/General.PNG?version=1&modificationDate=1302755792000

The latter diagram shows the basic steps a project goes through: fast test followed by sets and runs of acceptance tests, backup of production, acceptance tests, deployment, together with an instance of the process for a given commit through to live deployment.

Additional article:

http://jamesbetteley.wordpress.com/2011/05/27/continuous-delivery-using-build-pipelines-with-jenkins-and-ant/

Sui generis simple example:

http://blog.akquinet.de/2012/01/10/jumping-into-continuous-delivery-with-jenkins-and-ssh/

The key though is that it’s not just about running tests, it’s about letting users interact with the system, which is why we have the automatic vs. manual approval stages. We identify each step and use a tool like Jenkins to capture the whole workflow. After running through this process, a commit will have passed all tests, human and computer, and with a single button you can deploy to production.

Branching

  • Conventional wisdom says “don’t do it” (don’t use feature branches):

    • divergence is risky with increased likelihood of conflict with what has been worked on in mainline. When you switch off from your main conversation space (branch) you run the risk that you might have to do nasty re-integration later.

    • It complicates the whole deployment pipeline view. There is no way you are going to deploy a feature branch to production.

    • I am not against branching, however.

  • Feature toggles, “branching by abstraction”

    • Put your code in there, but turned off by a “switch” (say, a Drupal variable) so that the deploy does not have to show the new feature, but it can be toggled on and off at will. Facebook says it has its next six months of code running on production right now, deployed, masked from the users via feature toggles.

    • These are ways of staying in the same tight conversation space and thereby avoiding divergences.

  • Ponder on what this means. Really.

    • What changes would you actually have to make in your organization if you did not use branches. The continuous delivery gurus are saying that in order to have good software development processes we need to keep our conversations focused and contained. Letting people wander off into their own feature branch domain not necessarily the best idea.

[Victor Kane: I think that in a site building context (as opposed to the Drupal building context) there should be dev, test, staging and live branches in the repo, and that devs should be encouraged to have numerous local topic branches; and that this in no way contradicts the need to keep a tight conversation space in the devops pipeline, since devops should be seen as part and parcel to the actual merging and deployment of test, staging and prod branches. The Jenkins automation will then fit in perfectly as a superset of that merging and deployment, with no conflict. As Sam said “There is no way you are going to deploy a feature branch to production”.]

DIWD

Many have started with Drupal as hobbyists, and a deployment pipeline is insane for someone coming from that background. As a result many of us slide into much simpler workflows.

Jenkins in the middle

  • Everyone needs Jenkins. Period.

    • Start by moving every devopsy process you have into it (cron jobs, etc.). Jenkins will function as a state dashboard for your whole company/project.

    • Set up notifications. You can connect with jabber, and various bots, send emails, creating a rich information environment for your team broadcasting “this is the state of our code” within the at risk grey space between “done” and “done done”.

    • Plugins: http://jenkins-php.org

Build, Test, Deploy

  • Build

    • Takes what’s in version control and creates a deployable package

    • Ensures a deployment target exists

      • Create vhost (for example) if it doesn’t exist

    • As a discrete pipeline step, more pertinent for distros (drush make)

      • Creating binaries (not pertinent for Drupal)

      • Package (drush make, etc)

    • Build local environments

      • Tricky with Drupal

      • phing, ant; rake (pake, phake)

  • Build – Canonical Data

    • Environments need a canonical data source (usually a db dump)

    • Ponder on and document the when, how, and why for generating these artifacts

      • d.o. has daily sanitized dumps, for example

      • Jenkins can and should produce these artifacts for you! When it doubt put it in Jenkins

      • stage_file_proxy is the approach for files http://drupal.org/project/stage_file_proxy

        • don’t have to tranfer giga of assets in your files directory from production to every dev and test and staging instance.

        • needs some love but works fine

  • Deploy (Release)

  • Test

    • Pick your approach: TDD, BDD

      • Simpletest/PHPUnit, Selenium

        • Simpletest: many limitations

        • PHPUnit: external tests for clients, [unit tests] much better integration with Jenkins

        • Selenium for behavior driven development (BDD) [and black box functional testing]

      • Check out Capybara (unit tests), Cucumber (BDD) (ruby tools); Behat, Mink (newer projects)

      • [Victor Kane: TDD and BDD are not alternatives to pick from, rather both must be used, TDD for code unit tests and user acceptance tests for user stories, and BDD as an elaboration for feature/user story based functional testing: all of this needs to be done, that is, if you use BDD, you still have to have unit tests as part and parcel of implementation]

      • Time & scheduling

        • For example, test suite that takes 3 hours, shouldn’t be triggered automatically every five minutes 🙂

        • Smoke tests (no more than 10 minutes, preferably less than 5) just to make sure nothing is on fire (hence smoke…)

        • Test suite

          • Smoke tests

          • Commit continues and goes through hours long queue of tests

          • Queued up behind commits should not have their testing until the first batch is finished so you know which batch caused the test error

  • For all projects

    • Planning

      • Decide if you want to use branches

      • Sketch out your pipeline

        • Commits → testing → who is going to hit the green deploy button when Jenkins says all is cool?

        • Figure out the build/deploy strategy for each environment (dev included)

        • With Phing, ant, etc, you can include database shunting automation for dev, setting up test environments, etc.

    • The Repository

      • Just have only one repo

      • In top level subdirectories

        • Drupal

        • Tests

        • Build scripts

      • settings.php

        • They should be in version control!

          • settings.php ← in Git

          • settings.local.php ← not in Git (for each environment, db connection, memcache bins, cache settings etc.)

          • settings.$role.php ← in Git

  • More complex sites

    • If you have role-based machines, you need a configuration management layer on top of the layers described above for all sites.

    • Puppet, chef managing configuration management

    • Then you need local VMs

      • Vagrant! Ruby library for working with VirtualBox to set up and provision VMs, so you can provision local dev the same as production boxes, a real “production environment” you can safely break

      • Way beyond the same files and db, you have the same environment (MySql version, PHP version…)

      • drupal.org done like this (minute 46:30 on video)

  • Remember: GOOD ENOUGH

    • reams of stuff written on continuous delivery: read it, research it, discuss with Sam Boyer

    • but pick out the pieces you can do most easily and stick with what’s good enough

  • Questions: minute 52:00