Five Things I didn't know about

I want to share some exciting things I'm only just finding out about (the “Develop, Deploy, Rinse, Repeat” continuous delivery cloud platform for Drupal, Symfony and PHP based projects) that look as if they might have a lot to do with folks finding a straightforward way of enabling a truly Lean process applied to website and web application projects. We'll cover five things I didn't know about

  1. The Standard Platform Workflow is just what modern, serious PHP, Symfony and Drupal developers might expect and can easily be set up for all team members.

  2. The Standard Platform Architecture is container based and scales tremendously well for most use cases.

  3. They don't use Varnish! They use CDNs (content delivery networks)!

  4. There's an Enterprise Platform with its own truly scalable architecture and unique benefits

  5. A chance to get a first-hand report from someone actually using the Enterprise Platform.

1) Standard Platform Workflow

Note: People have been asking me about for some time now. Frankly I haven't had much time to investigate in my usual “put it through its paces and write some articles” fashion. But I did ask Robert for a dev account and got some exposure to a month or so ago. And since then it's popped up a number of times, at DrupalCon Latin America, and since I've been back. I've learned a few things that really got my attention and really want to share, so when Commerce Guys Robert Douglass suggested sponsoring an article to make that happen now, I jumped at the opportunity (disclaimer :) ) .

So what would be the first thing any designer or developer needs to do when they join a team working on a project? Clone the current development branch (called, for example, Sprint 2) and get it working. Well, with the Standard Platform, the first thing you do is just that. And once you do you don't just get the code; you get the whole environment, visible to you as a working site.

And what's the second thing any team member needs to do once they find themselves assigned to any job of work? Well, create a feature branch, get it working, make changes, test them, then sync down changes other developers have made in the meantime before then merging in their own tested changes. Now, with, these are simply basic automated operations that form a natural part of the workflow action repertoire. So, first, when you create that feature branch, a matching environment is automatically created, so as work is performed and changes made, they can be run tested in that self-contained site. Once the work is complete and fully tested, you sync down changes others have made, test again, then merge up your work.

This concept of “sync down” and “merge up” is natural to anyone using development best practices on the basis of a distributed version control system like Git. It has nothing to do with per se. Which is why when you join a team using everything works exactly as modern, serious designers and developers would expect. That's because in you are not limited to any hard-coded scheme created a priori. You create the hierarchy of branches and matching environments as needed, and name them anything you like. When you first create a project, you start off with the master branch, just like on GitHub. The corresponding running environment is what will be the live production site, once it's enabled. But then, in terms of child branches, it's totally up to the team how to proceed. You can create a child branch for each Sprint, and children for each developer, and they can make their own feature, topic and bug fixing branches + environments at will. Or any other scheme that naturally matches your adopted process, or the special characteristics of the project.

Not only that, but the project owner or whoever is managing the project can create different sets of permissions over actions for each branch and their children. Where appropriate, read only permissions could be granted, or full permissions but only on a given sprint branch or only on the individual developer branch under the sprint branches in which they will be participating. It's a very flexible system.

More examples, and the nuts and bolts of how this works and the details of the tools actually used, both on the command line as well as with the Web UI for each team member, can be found in the documentation overview. Plus Ori Pekelman of the Commerce Guys just published this great article How the PaaS works. I'm planning on doing more articles on drilling down to a non-trivial level of detail.

One of the main points I was trying to make at my DrupalCon Latin America presentation on Lean DevOps and UX for Drupal development was that Agile and Lean are just words, pure sophistry, when it comes to web development unless the walls isolating the various discipline silos common to it can be broken down and the whole team can work in an atmosphere that encourages constant cross-collaboration. Now, for this to happen, there has to be a real-world and fully functional Lean UX and DevOps process in place. Without a smooth, tailored provisioning process in place for the whole team, there won't really be cross-collaboration. And without continuous build and delivery workflows tailored to special team and project needs, knowledge sharing and constant feedback and frequent user and client validation (without which Lean degenerates to just a buzzword and Waterfall rears its ugly default head) just will not happen. As I say, I plan on testing this all out with real world examples I'll be publishing here, but it does look like goes a long way towards providing all of this right off the shelf, for a variety of (for the present only) PHP based toolstacks (Symfony, Drupal, more on the way).

Apart from the full-stack environment for every Git branch, developers can build sites on the server with a combination of Drush Make and your own custom code using a dedicated REST API and CLI tool support, which also allows for full continuous integration automation. Again, more articles to follow as these features are put through their paces!

2) Scaling with the standard container based architecture

The development process briefly described above certainly sets apart as being different. But what about performance? Well, the architecture is based on micro-containers for each system component. For example, the Http server, a database instance, or in memory object store or Solr, for example, are each micro-containers. This architecture supports very flexible and straightforward scaling by simply plugging in additional micro-containers as needed.

Then, if that proves insufficient, what really sets apart is the option to migrate to an exclusive enterprise architecture (see point 4 for details). While I haven't had a chance to test out the scaling and to carry out tests of my own on Standard Platform performance, Robert also gave me a demo of performance on the enterprise architecture, profiling real-world high traffic sites, all of which gave response times of under one second without exception. To be used when 3 second page loads are unacceptable.

3) CDNs instead of Varnish

To be honest I was really curious about this one. Everyone and his cat uses Varnish. As part of the demo I requested I took the opportunity of asking about this. The explanation I received is that CDN is actually the best choice as HTTP cache, because it is made the port of entry for all requests. Using Cloudflare by default and Fastly in some cases, sub-100 ms speeds are a matter of course, with 45 edge locations all around the world. This is extremely innovative, and I owe it to myself to take a closer look at this one also. The timing is perfect too, what with Varnish putting all of Drupal-land into a tizzy recently when it bifurcated its project into the paid Varnish Plus, on the one hand, and the Community edition (shades of Fedora / Red Hat?), on the other. In any case, the CDN HTTP fronting certainly saves us from the royal pain of setting it up with Drupal, as well as from the high rate of non-trivial anomalies associated with it when it is working.

4) Enterprise platform architecture and benefits

When single-second page loads and true high-availability and Zero-downtime scalability are a must for mission critical sites, we can't go on with business as usual and one size fits all. But as a matter of course. Because in most cases, a costly special custom engineering effort is becomes necessary. Here we are talking about the option of an off-the-shelf true enterprise deployment solution. And before I saw the demo, I truly had no idea such a thing existed.

What I didn't know is that the architecture for Enterprise Platform is not container based at all, since that would simply mean additional and unwarranted overhead and deployment updating, once we are fully aware of the scope of scalability required. As mentioned before, the DNS setup routes all HTTP requests to the CDN being used, from which it is handed off to an Elastic Load Balancer which distributes the traffic after performing a health check on available Nginx/PHP-FPM based Drupal (or other app) instances (three by default). Further load balancing distributes DB queries over three MariaDB instances in the form of a Galera cluster (you learn something new every day: Writes go to a single Master. What's interesting is that all three instances are synchronous Masters, with the Load Balancer electing which instance carries the role of Master in case of failure.

The default of three stack instances can be scaled out to n instances as needed.

From there the architecture is rounded out by a single Solr Cloud and an SSD-based network file system.

And in the demo, I was able to confirm first-hand less-than-a-second page loads on actual running sites.

OK, can't wait to get my hands on that! Large-scale project improvements anybody?

5) First-hand account of what it's like to use the full-support high availability Enterprise platform

The benefits of worry-free seamless scalability, from highly performance container based architecture, to truly high availability enterprise architecture using the same development and deployment platform interface are impressive.

While I was at DrupalCon Latin America I actually got the chance to speak to Yeny Paola Mariño Parra, Systems Manager for Editorial La Patria S.A. in Colombia, who have recently migrated their 3-million page-views a day media-rich online newspaper to the Enterprise Platform. Even though it was made clear from the start that their experience with is only a few months old, she was kind enough to share her experience with me:

“Once the site was developed, it went live on our own dedicated servers. We were satisfied with the results. But we immediately saw we had a problem: We just don't have time to be fiddling with server configurations and scaling problems, and the site was taking up all our time and then some. We needed a quick solution which wasn't going to break our budget,” Yeny explained. “And one of our biggest problem as we searched for solutions was that nobody understood what we were looking for, regular hosting had no special knowledge of Drupal and its specialized needs, and failed to understand the fact that we had an extremely tight schedule as well as a limited budget and a specialized workflow, and that we simply could not be left to our own devices. We needed understanding and technical insight and help. And that is what interested us in”

What La Patria needed was nothing less than migrating their deployment target to a proper platform that would relieve them of all worry and concern, respect their workflow, and at the same time do it all in a month. And it was nothing less than the Christmas holiday season, and the plan called for them to prepare, migrate, learn how to transplant their editorial workflow and become productive in time for the early January 2015 launch date. understood their situation and made a reasonable proposal that La Patria could grow into as they learned and became familiar with the platform over time. The newly migrated site, and the daily workflow, launched on time.

“The folks at showed superior technical knowledge and understood our special needs and workflow immediately. They gave us excellent advice and provided a support team that also realized on a human level that there were real people behind our project, despite the geographical distance. Real people with a tight schedule, and with limited resources.”

I'm looking forward to checking in with Yeny and the people at La Patria to see how that goes.

Knowing that you have a true high availability enterprise architecture option open and being able to migrate to that option with full support whenever growth makes that necessary will surely bring peace-of-mind to many companies and organizations.

At the risk of repeating myself, I can't wait to try all that out on a project soon.

Please use the hashtag #FiveThingsAboutPlatformSH with any questions or comments!