Problem of teams needing to merge databases

There is an interesting thread at the Wrox P2P Forum on how to merge developoment, configuration and content efforts among teams working with multiple database for the same site.

See http://p2p.wrox.com/book-leveraging-drupal-getting-your-site-done-right-...

The two questions posed are:

Currently, I'm in chapter 3, and I have a couple of questions about the way you've handled storing a dump of the database in the repository.

1. What happens if you have multiple developers working on a project and they each install different modules? Won't there be a conflict between their database dumps?

2. What happens when you've deployed the site and the end user is happily adding content, but you continue to develop the site? Later down the road, your database will have new module configurations, but the production database will have lots of new content. How do you merge the configuration part of the development into a live site without loosing data?

Excellent questions.

My answer:

You hit the nail on the head

First of all, thanks so much for your praise. Basically, in writing the book I have tried to achieve just that, the possibility of folks not having to go through a steep learning curve by putting everything you need in a single place.

Your two questions point to a huge difficulty in Drupal architecture, namely that of the configuration, both in terms of things like blocks and module installation, being mixed with content in the database in such a way that it is no trivial task to "merge" similar databases in the way you describe.

On the one hand, there are possible solutions, that need to be digested and adapted to your process. On the other, there are integration and build strategies you must set up in your team.

Regarding possible solutions, every now and then in the Drupal community, solutions have been promised, but there has never been a straightforward, practical solution that just works.

Reading and ways forward on this topic:

1. http://ceardach.com/blog/2009/01/lea...and-deployment
2. Also see Kathleen's Database scripts, published on Drupal, which may very well be the answer to your question, but it needs to be digested and adapted to your process: http://drupal.org/project/dbscripts;
3. Greg's deploy module: http://drupal.org/project/deploy
4. The Aegir group of modules, with hosting, provisioning and site install/deploy scripts: http://groups.drupal.org/aegir-hosting-system
5. Other solutions that have been suggested over the years, like incrementing the node counter by odd or even increments on development and production machines, and the like.
6. Installing content remotely via protocols such as REST and XML-RPC (blog api and the services module) allow some separation of configuration and content. See my own article: http://awebfactory.com.ar/node/297

So, the other side of the coin is that a team strategy has to be developed, and it involves an "integration moment". That is, individual developers, who may use the SVN repository to branch, must each be invited to merge.

This may not be as painful a moment as might be imagined. By branching the code, that is, the Drupal instance, a developer may be able to abstract their work into a module, including code-based views as .inc files (dealt with in Chapter 14), where the database configuration can be dealt with as an install script that accompanies the module. Installation profile resources (also dealt with in Chapter 14) are another method, especially taking into account modules that export database configuration to scripts.

This is as realistic an answer as I can give you at this point in Drupal history.

On a practical level, you may simply need more than ever to work on a test site, figure out what you want to do, then repeat that work on the production site where the content is held.

We can explore this in more detail, as need be, and you are invited also to join in the Leveraging Drupal Workshop Central, by leaving comments, which is just getting underway (there is a separate child page for each chapter): http://awebfactory.com.ar/node/348#workshop

I will place the gist of this discussion under Chapter 3.

Thanks again for your feedback.

Victor Kane

At least there is hope

This stuff is really new to me so it's going to take some "digesting" like you said.

I like the idea of abstracting development into a module. I'm not there yet, but I'm working on it :)

From the sound of Kathleen's article it seems it is possible to dump the user data separately. That, I would think, would be even more important than merging development configuration as I anticipate running into the user data issue more frequently.

So, are you wanting this site, as opposed to the forum, the primary place for discussions? Thanks for your help :)

Right, let's see how this develops

Looking forward to some concrete problems/examples/solutions...

As to the two sites, this site is great for a chapter by chapter, topic- and detailed-oriented workshop approach, where we get down and dirty and it's easy to find stuff afterwards; which is to say, how we use the book; as opposed to discussion strictly about the book itself, which is the topic of the Wrox forum.

Looking forward to dialog either way!

Victor Kane