Setting up Cloud App Server with CouchDB, Node.js and Express on Ubuntu 10.04 LTS Part I

In this series of articles we'll see how to set up a Cloud App Server with CouchDB and Node.js on Ubuntu 10.04 LTS, and why.

The Web App is dead! Long live the web app

It's not the rise and fall of the meca of native mobile apps. I think they're beginning to be dead too, if left to themselves. I guess Quora is a good example. It's a software application designed with an architecture that simply cofins the old single server or cluster web application architecture served up in the browser. As explained in Phil Whelan's exciting article Quora's Technology Examined, it was created from the ground up as a cluster of distributed servers setting up a totally asynchronous dialog with the browser client in order to fulfill a very simple set of user stories, and fulfill them it does.

Distributed components: the signature of the cloud app

Without getting involved in language or platform wars, it is clear that the starting point is a simple set of requirements forming a killer app, for which the heavens are assaulted to yield constellations of servers to get you implementation.

Distributed, schema-less databases

First off, there is no such thing as THE database. And I'm not talking about master slave replication or clustering of THE database. Now, data is distributed among the client, database servers, server memories (with technologies like Thrift), according to what's really going on. Quora uses MySQL but without joins, sharded and schema-less in the How FriendFeed uses MySQL to store schema-less data manner.

As a veteran of document based storage (I built a newspaper CMS based on Xindice, a native XML database, about eight years ago) I'm gonna go with CouchDB. I know a lot of people are going with MongoDB, another popular no-SQL, document-oriented  database. They are both good, depending on what you are doing. My choice in favor of CouchDB right now, curiously enough, originated in a blurb on the MongoDB site itself Comparing Mongo DB and Couch DB:

if we were building Lotus Notes, we would use Couch as its programmer versioning reconciliation/MVCC model fits perfectly.  Any problem where data is offline for hours then back online would fit this.  In general, if we need several eventually consistent master-master replica databases, geographically distributed, often offline, we would use Couch.

Mobile Couchbase, as one example:

Couchbase Mobile allows developers to write web applications one time, synchronize multiple copies and share data and applications across any computing platform or mobile device.... keeps the data on all your devices synchronized with the Couchbase database in the cloud.

That sounds very exciting.

And did I mention that CouchDB is fault tolerant?

Distributed protocol servers

Of course one of the things we desperately need to do is to liberate ourselves from data-centric architectural slavery. We wanna go to a very simple set of user stories and get it done. The above quora example shows that a modern cloud app isn't going to shove everything down a single web server bottleneck at all; apart from the traditional load balancers and reverse proxy servers, it also has:

  • A lightweight MVC web framework, divided between page generation and live update handling
  • A separate comet (long-polling) server
  • Messaging clients and servers (Thrift)

Again, without going to the language wars, I'm gonna go with Node.js

"If you want to make something where the user gets a real-time update stream of some kind, you need to keep open a bunch of connections to the server. That's very expensive in, say, a PHP architecture. But in Node, it's nearly free. You can keep the connections open for a very long time, and the incremental cost per connection is very low," Ranney, Voxer's CTO, tells The Register.

And with Express, "High performance, high class web development for Node.js".

I've been working for some time with CouchDB and Node.js, and taking a good hard look prototyping stuff in Express, so I was super gratified, not to say super positive feedbacked, as well as re-assured, to learn that old friends are basing some super Open Source software on these technologies as well. So it's all very, very exciting. So let's get started.

Installing CouchDB

Like Quora, let's just go with Ubuntu as server OS, whether dev, staging or live. To start off my staging server(s), I just headed over to linode (link top right of this page) to grab a VPS (live will of course go to the cloud, but Rackspace looks much more likely now than Amazon...). I selected the 10.04 LTS as a matter of principle (following wise suggestions) since I will be able to update for years to come.

If you've never used a VPS before and are coming from shared hosting web app environments, be sure to check out the Linode Getting Started page or similar if you are using a different setup, in order to log in via ssh, set up the hostname and timezone, etc.

So, now we have a brand-new server, how to install CouchDB?

The CouchDB page itself is actually the best place to go for resources of all kinds, including the documentation Wiki. There is an installation page there for installing on just about any platform, and a specific Ubuntu installation page giving various alternatives for installing on Ubuntu:

  • From the Ubuntu software repository via aptitude or apt-get (apt-get install couchdb)
  • Using the Ubuntu installer from the good folks at CouchOne
  • Compile from source

While the second option will give you the latest stable version with all libraries and dependencies included, and may be the best option for many people, I decided it would be best for maintenance purposes and just in the name of full control and independence to go with compiling from source with everything in /usr/local for easy removal and reinstallation. That's what I will document here.

Several examples are mentioned and while going with the minimal install (Example 2) was tempting and might be the way to go eventually, I am gaining familiarity and intimacy for now, so Example 1 (Alternative) ("This set of install instructions is modified from Example 1 and places all current CouchDB files within the /usr/local/ tree. Any other CouchDB related files outside of /usr/local/ can be deleted or ignored.") is my choice for now.

Many thanks to http://tommy.chheng.com/2009/07/10/installing-couchdb-on-ubuntu-problems... for the addition of unmet dependencies before configuring, compiling and installing couchdb.

These same dependencies are mentioned for Example 2.

Here is Example 1 (Alternative) with my own modifications to make it work with 1.0.2, and comments in ###:

sudo su
# Install dependencies required to build couchdb from source
### Huge number of items, takes a few minutes
apt-get build-dep couchdb
###### Additional dependencies (taken from Example 2)
### Add add-apt-repository command:
apt-get install python-software-properties
### Add spidermonkey repo (nobody got this right, but these steps work!)
add-apt-repository ppa:launchpad/ppa
apt-get update
apt-get upgrade
apt-get install libmozjs-dev
### Add more stuff as per example 2
apt-get install libicu-dev libcurl4-gnutls-dev libtool erlang-dev erlang
###### end additional dependencies
cd /opt # download the latest release from http://couchdb.apache.org/downloads.html
### I went to http://couchdb.apache.org/downloads.html
### Clicking on the latest and greatest most stable release took me to
### http://www.apache.org/dyn/closer.cgi?path=/couchdb/1.0.2/apache-couchdb-...
### This suggested mirror sites. My download <url> was
### http://apache.dattatec.com//couchdb/1.0.2/apache-couchdb-1.0.2.tar.gz
# wget <url> tar xvzf apache-couchdb-x.xx.x.tar.gz
### So for me: cd apache-couchdb-1.0.2
cd apache-couchdb-x.xx.x # see xulrunner directions below and do them this fixes problem where basic test suite fails to run. # Note: To check what XULRunner version you have installed use xulrunner -v
### I did the xulrunner -v
### Which gave me:
### Mozilla XULRunner 1.9.2.17 - 20110424212015
### So my configure statement was
### ./configure --prefix=/usr/local/ --with-js-lib=/usr/lib/xulrunner-devel-x.x.x.x/lib --with-js-include=/usr/lib/xulrunner-devel-1.9.2.17/include ./configure --prefix=/usr/local/ --with-js-lib=/usr/lib/xulrunner-devel-x.x.x.x/lib --with-js-include=/usr/lib/xulrunner-devel-x.x.x.x/include # Test installation, you will have problems regarding libmozjs and xulrunner here if you have not already corrected them ### This actually takes quite a while but it is so nice to know you have everything right and see that cool "Results: PASS"
make check # Now you can compile and install couchdb
### lightning fast!
make && make install
### Make file does NOT create couchdb user as stated # vi /etc/passwd here and change home directory to /usr/local/var/lib/couchdb/, couchdb user is created during make && make install ### Let's create couchdb user manually as per Example 2
useradd -d /usr/local/var/lib/couchdb couchdb

### This is right # change file ownership from root to couchdb user and adjust permissions chown -R couchdb: /usr/local/var/{lib,log,run}/couchdb /usr/local/etc/couchdb chmod 0770 /usr/local/var/{lib,log,run}/couchdb/ chmod 664 /usr/local/etc/couchdb/*.ini chmod 775 /usr/local/etc/couchdb/*.d # start couchdb cd /etc/init.d ln -s /usr/local/etc/init.d/couchdb couchdb /etc/init.d/couchdb start # Start couchdb on system start update-rc.d couchdb defaults # Verify couchdb is running curl http://127.0.0.1:5984/ # {"couchdb":"Welcome","version":"1.0.1"}
###### OMG! Up and running!
# cd
# /etc/init.d/couchdb start
* Starting database server couchdb
...done.
# update-rc.d couchdb defaults
Adding system startup for /etc/init.d/couchdb ...
/etc/rc0.d/K20couchdb -> ../init.d/couchdb
/etc/rc1.d/K20couchdb -> ../init.d/couchdb
/etc/rc6.d/K20couchdb -> ../init.d/couchdb
/etc/rc2.d/S20couchdb -> ../init.d/couchdb
/etc/rc3.d/S20couchdb -> ../init.d/couchdb
/etc/rc4.d/S20couchdb -> ../init.d/couchdb
/etc/rc5.d/S20couchdb -> ../init.d/couchdb
# curl http://127.0.0.1:5984/
{"couchdb":"Welcome","version":"1.0.2"}
#
###### end OMG! Up and running!
# run test suite in firefox at http://127.0.0.1:5984/_utils/couch_tests.html?script/couch_tests.js
### This won't work at all, because Futon is only available at this point on the "local" machine
### and we have a server and we don't want folks taking a nap on our futon!
### ... so see next section, on how to easily set up a secure tunnel to run futon locally on a dev machine
### ... and on how to set up remote access

Securely Admin CouchDB with an SSH Tunnel

See http://library.linode.com/databases/couchdb/ssh-tunnel

I have attached the perl script couchdb-tunnel.pl found on that page, edited the configurate found in the first two lines (lines 3 and 4 left at 5984 and 127.0.0.1 respectively), and run it on both Ubuntu desktop and Mac Pro, and obtained the following results:

Secure administration of CouchDB using an ssh pipe

Setting up remote access

In a manner analogous to MySql, by default CouchDB is set up for local access only, through 127.0.0.1 . You can change this just as easily, by editing the file /usr/local/etc/couchdb/local.ini and setting bind_address to 0.0.0.0

Now you can have apps going on with design deocuments, etc.

However, this will make your server insecure, and you will need to start using authentication and authorization for administration and editing. Check out the lower right-hand corner of Futon, where it says "Welcome to Admin Party! Everyone is admin. Fix this (link)" to get started.

So CouchDB is installed, now wot?

Couple of cool resources to get you started:

Installing Node.js

The following pages give good background on installing node.js and npm in various situations, including without root access and even without sudo access:

In any case, I do want to install as root, making the install default simple and server wide, and maintainable. So I chose accordingly.

Dependencies for installing node.js

"Node.js itself has no external dependencies except common build tools as well as pythons for the build system itself. On OSX you must install XCode for this to work, and on Ubuntu you probably have to run: $ apt-get -y install build-essential" - Felix's Node.js Beginner's Guide http://nodeguide.com/beginner.html#installation

 

  • build-essentials
  • curl
  • openssl (libssl-dev)
  • python for node's build tools
  • git

All set to go on these after installing CouchDB as per above steps. To install git:

### So the following is not necessary unless starting out fresh from this point.
# apt-get install build-essential curl libssl-dev
### Install latest git from source
# cd /opt
# apt-get remove git-core
# apt-get build-dep git-core
### Went to http://git-scm.com/download and got the link to the source for latest stable version
# wget http://kernel.org/pub/software/scm/git/git-1.7.5.2.tar.bz2
# tar xvjf git-1.7.5.2.tar.bz2
# cd git-1.7.5.2
# ./configure
# make
# make install
# cd
# git --version
git version 1.7.5.2

Installing node.js

Note: This is actually the most straightforward way, once the dependencies have been dealt with:

Go to http://nodejs.org/#download and grab the tarball (do this in /opt or wherevery you like):

$ wget http://nodejs.org/dist/node-v0.4.8.tar.gz
$ tar xvzf node-v0.4.8.tar.gz
$ cd $ cd node-v0.4.8
$ ./configure
$ sudo make install

(See https://github.com/joyent/node/wiki/Installation for installing locally, relative to your $HOME)

Installing latest stable version from git as root:

# cd /opt
# git clone git://github.com/joyent/node.git
# cd node
# git checkout origin/v0.4 # optional. Note that master is unstable. Ignore 'detached HEAD'...
# Alternative (check out tag): $ git checkout v0.4.8
# ./configure
# make install
# which node
/usr/local/bin/node

Now test the node console:

 node
> var foo = {bar: 'baz'};
> console.log(foo);
{ bar: 'baz' }
> .help
> .exit

Edit: Very important to add checkout statement for the latest stable version, instead of just Master unless you are experimenting. To find out which is the current most stable version, go to http://nodejs.org/#download and to see. At the time of writing, it was node-v0.4.8.tar.gz, which can be installed by unpacking and running configure and make install as detailed above.

Installing npm (Node.js package manager)

# curl http://npmjs.org/install.sh | sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 3874 101 3874 0 0 41540 0 --:--:-- --:--:-- --:--:-- 65661
fetching: http://registry.npmjs.org/npm/-/npm-1.0.6.tgz
0.5.0-pre
1.0.6
prefix=/usr/local

This script will find and eliminate any shims, symbolic
links, and other cruft that was installed by npm 0.x.

Is this OK? enter 'yes' or 'no'
yes

All clean!
! [ -d .git ] || git submodule update --init
node cli.js cache clean
node cli.js rm npm -g -f --loglevel error
node cli.js install -g -f
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm.js
node-uuid@1.1.0 /usr/local/lib/node_modules/npm/node_modules/node-uuid
npm@1.0.6 /usr/local/lib/node_modules/npm
It worked
# which npm
/usr/local/bin/npm

So node.js is installed, now wot?

Installing Express

According to instructions at https://github.com/visionmedia/express I did the following in order to access the express executable globally:

# npm install -g express
/usr/local/bin/express -> /usr/local/lib/node_modules/express/bin/express
connect@1.4.1 /usr/local/lib/node_modules/express/node_modules/connect
mime@1.2.2 /usr/local/lib/node_modules/express/node_modules/mime
qs@0.1.0 /usr/local/lib/node_modules/express/node_modules/qs
express@2.3.7 /usr/local/lib/node_modules/express

Then, as a dev user, I create a first create a skeleton

$ express testapp
create : testapp
create : testapp/app.js
create : testapp/views
create : testapp/views/layout.jade
create : testapp/views/index.jade
create : testapp/public/stylesheets
create : testapp/public/stylesheets/style.css
create : testapp/public/images
create : testapp/public/javascripts
create : testapp/logs
create : testapp/pids
create : testapp/test
create : testapp/test/app.test.js

Then I cd into it and install express and jade dependencies

$ cd testapp
~/testapp$ npm install express jade
jade@0.11.0 ./node_modules/jade
mime@1.2.2 ./node_modules/express/node_modules/mime
connect@1.4.1 ./node_modules/express/node_modules/connect
qs@0.1.0 ./node_modules/express/node_modules/qs
express@2.3.7 ./node_modules/express

Then I can execute the test app:

$ node app.js 
Express server listening on port 3000

And point my browser at http://my-vps-server.com:3000/

 

Express

Welcome to Express

 

Installing additional components

[tbd]

AttachmentSize
couchdb-tunnel.pl_.txt1.47 KB
Screen shot 2011-05-22 at 12.29.25 PM.png438.82 KB

http://www.google.com/trends?

http://www.google.com/trends?q=drupal just wanted to point out that it seems to be stagnating, and I expect it to go down. Drupal was better than its competitors, but it's not a framework and real developers use real frameworks. Also not sold on Couch or even Cassandra for that matter. Until this technology matures, and they can show that they are somehow actually better I am not switching from an RDMS. Reddit has shown that in Amazon's cloud Cassandra is not stable. Twitter refused to switch from mysql for its major store. So I think sticking with RDMS is a safe bet going forward, there may come an inflection point but I am not seeing it yet.

Also, I am becoming a fan of real PHP frameworks and perhaps you should look into them too, I suggest Symfony or the lighter Kohana. Express and node are definitely the future. Drupal is the past, at least for me... I am sure it has enough momentum to carry it for a few more years, but I believe it's over. Ruby also has momentum that I don't think is going away; though I hate the language compared to Python.

Thanks for the writeup... good to know everyone is keeping up and is ready for the future!

Capitalist crisis not providing tools equal to the task

The core of the problem is that while enterprise apps are demanding more and more (also fueled by the crisis), none of the current capitalist business models, not even "open source", is capable of providing the necessary solid tools.

Drupal is trending down, but is not stagnating. It is still the best tool for many tasks; but as I write in my article, all web apps and their so-called frameworks (you are right there) are dead. So even if Drupal owned up to its code debt and wrong data-centric architecture, it cannot change the objective limitation of web apps themselves.

They are dead because enterprise apps, and as a result, users, are getting ready for much more; for distributed, real-time, global apps. But because of the limitations of how large corporations function in the decline of capitalism, no solid tools are coming out, no obvious answers are being made, no silver bullets paraded. No more "the C programming language"; no more Cobol; no more Java; no more C++.

So we have to go with what we can get our hands on. And make it work with spit and grime.

I think that is the spirit of Node.js, but in any case, lean and mean and scalable seems the way to go and enterprising pioneers can make it happen. Single thread event loop plus Sinatra in SSJS seems like a fine hammer and anvil.

Also, fortunately, it's very easy to try CouchDB and then swap it out for say MongoDB or others. But a wait and see attitude makes us part of the problem and not part of the solution.

In all of the course of my career, I have always gone cutting edge. The future requires progress, the capitalist anarchy has got to go, and we have to have tools in place for people to live and work.

On the question of RDBMS, note my reference to the noSQL use of MySQL documented by FriendFeed, it is one way to go, basically making use of MySQL storage without the relational part. It is RDBMS without the "R",

Victor

Nice tutorial and excellent

Nice tutorial and excellent Node Resources you have pointed out here.
We have talked while ago about drupal and other stuff but the combination of Node, Express and CouchDB has left the old things quite behind. And I personally never had given Javascript the attention it deserved since Netscape and its SSJS.
Cheers !

Yes, SSJS has reappeared with a vengeance

Thanks, Aldo, for your comment. I will certainly be recounting my adventures here. Drupal is still best of breed in terms of a web application CMS Framework, if that meets requirements, but the point of urgency, need and demand does seem to be shifting, as it has always shifted...

Victor