Setting up Cloud App Server with CouchDB, Node.js and Express on Ubuntu 10.04 LTS Part I

In this series of articles we'll see how to set up a Cloud App Server with CouchDB and Node.js on Ubuntu 10.04 LTS, and why.

Posted on: Last updated on: Written by:

In this series of articles we’ll see how to set up a Cloud App Server with CouchDB and Node.js on Ubuntu 10.04 LTS, and why.

The Web App is dead! Long live the web app

It’s not the rise and fall of the meca of native mobile apps. I think they’re beginning to be dead too, if left to themselves. I guess Quora is a good example. It’s a software application designed with an architecture that simply cofins the old single server or cluster web application architecture served up in the browser. As explained in Phil Whelan’s exciting article Quora’s Technology Examined, it was created from the ground up as a cluster of distributed servers setting up a totally asynchronous dialog with the browser client in order to fulfill a very simple set of user stories, and fulfill them it does.

Distributed components: the signature of the cloud app

Without getting involved in language or platform wars, it is clear that the starting point is a simple set of requirements forming a killer app, for which the heavens are assaulted to yield constellations of servers to get you implementation.

Distributed, schema-less databases

First off, there is no such thing as THE database. And I’m not talking about master slave replication or clustering of THE database. Now, data is distributed among the client, database servers, server memories (with technologies like Thrift), according to what’s really going on. Quora uses MySQL but without joins, sharded and schema-less in the How FriendFeed uses MySQL to store schema-less data manner.

As a veteran of document based storage (I built a newspaper CMS based on Xindice, a native XML database, about eight years ago) I’m gonna go with CouchDB. I know a lot of people are going with MongoDB, another popular no-SQL, document-oriented  database. They are both good, depending on what you are doing. My choice in favor of CouchDB right now, curiously enough, originated in a blurb on the MongoDB site itself Comparing Mongo DB and Couch DB:

if we were building Lotus Notes, we would use Couch as its programmer versioning reconciliation/MVCC model fits perfectly.  Any problem where data is offline for hours then back online would fit this.  In general, if we need several eventually consistent master-master replica databases, geographically distributed, often offline, we would use Couch.

Mobile Couchbase, as one example:

Couchbase Mobile allows developers to write web applications one time, synchronize multiple copies and share data and applications across any computing platform or mobile device…. keeps the data on all your devices synchronized with the Couchbase database in the cloud.

That sounds very exciting.

And did I mention that CouchDB is fault tolerant?

Distributed protocol servers

Of course one of the things we desperately need to do is to liberate ourselves from data-centric architectural slavery. We wanna go to a very simple set of user stories and get it done. The above quora example shows that a modern cloud app isn’t going to shove everything down a single web server bottleneck at all; apart from the traditional load balancers and reverse proxy servers, it also has:

  • A lightweight MVC web framework, divided between page generation and live update handling
  • A separate comet (long-polling) server
  • Messaging clients and servers (Thrift)

Again, without going to the language wars, I’m gonna go with Node.js

“If you want to make something where the user gets a real-time update stream of some kind, you need to keep open a bunch of connections to the server. That’s very expensive in, say, a PHP architecture. But in Node, it’s nearly free. You can keep the connections open for a very long time, and the incremental cost per connection is very low,” Ranney, Voxer’s CTO, tells The Register.

And with Express, “High performance, high class web development for Node.js“.

I’ve been working for some time with CouchDB and Node.js, and taking a good hard look prototyping stuff in Express, so I was super gratified, not to say super positive feedbacked, as well as re-assured, to learn that old friends are basing some super Open Source software on these technologies as well. So it’s all very, very exciting. So let’s get started.

Installing CouchDB

Like Quora, let’s just go with Ubuntu as server OS, whether dev, staging or live. To start off my staging server(s), I just headed over to linode (link top right of this page) to grab a VPS (live will of course go to the cloud, but Rackspace looks much more likely now than Amazon…). I selected the 10.04 LTS as a matter of principle (following wise suggestions) since I will be able to update for years to come.

If you’ve never used a VPS before and are coming from shared hosting web app environments, be sure to check out the Linode Getting Started page or similar if you are using a different setup, in order to log in via ssh, set up the hostname and timezone, etc.

So, now we have a brand-new server, how to install CouchDB?

The CouchDB page itself is actually the best place to go for resources of all kinds, including the documentation Wiki. There is an installation page there for installing on just about any platform, and a specific Ubuntu installation page giving various alternatives for installing on Ubuntu:

  • From the Ubuntu software repository via aptitude or apt-get (apt-get install couchdb)
  • Using the Ubuntu installer from the good folks at CouchOne
  • Compile from source

While the second option will give you the latest stable version with all libraries and dependencies included, and may be the best option for many people, I decided it would be best for maintenance purposes and just in the name of full control and independence to go with compiling from source with everything in /usr/local for easy removal and reinstallation. That’s what I will document here.

Several examples are mentioned and while going with the minimal install (Example 2) was tempting and might be the way to go eventually, I am gaining familiarity and intimacy for now, so Example 1 (Alternative) (“This set of install instructions is modified from Example 1 and places all current CouchDB files within the /usr/local/ tree. Any other CouchDB related files outside of /usr/local/ can be deleted or ignored.”) is my choice for now.

Many thanks to http://tommy.chheng.com/2009/07/10/installing-couchdb-on-ubuntu-problems-and-fixes/ for the addition of unmet dependencies before configuring, compiling and installing couchdb.

These same dependencies are mentioned for Example 2.

Here is Example 1 (Alternative) with my own modifications to make it work with 1.0.2, and comments in ###:

sudo su
# Install dependencies required to build couchdb from source
### Huge number of items, takes a few minutes
apt-get build-dep couchdb
###### Additional dependencies (taken from Example 2)
### Add add-apt-repository command:
apt-get install python-software-properties
### Add spidermonkey repo (nobody got this right, but these steps work!)
add-apt-repository ppa:launchpad/ppa
apt-get update
apt-get upgrade
apt-get install libmozjs-dev
### Add more stuff as per example 2
apt-get install libicu-dev libcurl4-gnutls-dev libtool erlang-dev erlang
###### end additional dependencies
cd /opt
# download the latest release from http://couchdb.apache.org/downloads.html
### I went to http://couchdb.apache.org/downloads.html
### Clicking on the latest and greatest most stable release took me to 
### http://www.apache.org/dyn/closer.cgi?path=/couchdb/1.0.2/apache-couchdb-1.0.2.tar.gz
### This suggested mirror sites. My download <url> was 
### http://apache.dattatec.com//couchdb/1.0.2/apache-couchdb-1.0.2.tar.gz
# wget <url>
tar xvzf apache-couchdb-x.xx.x.tar.gz
### So for me: cd apache-couchdb-1.0.2
cd apache-couchdb-x.xx.x

# see xulrunner directions below and do them this fixes problem where basic test suite fails to run.
# Note: To check what XULRunner version you have installed use xulrunner -v
### I did the xulrunner -v
### Which gave me: 
### Mozilla XULRunner 1.9.2.17 - 20110424212015
### So my configure statement was
### ./configure --prefix=/usr/local/ --with-js-lib=/usr/lib/xulrunner-devel-x.x.x.x/lib --with-js-include=/usr/lib/xulrunner-devel-1.9.2.17/include
./configure --prefix=/usr/local/ --with-js-lib=/usr/lib/xulrunner-devel-x.x.x.x/lib --with-js-include=/usr/lib/xulrunner-devel-x.x.x.x/include

# Test installation, you will have problems regarding libmozjs and xulrunner here if you have not already corrected them
### This actually takes quite a while but it is so nice to know you have everything right and see that cool "Results: PASS"
make check

# Now you can compile and install couchdb
### lightning fast!
make && make install

### Make file does NOT create couchdb user as stated
# vi /etc/passwd here and change home directory to /usr/local/var/lib/couchdb/, couchdb user is created during make && make install
### Let's create couchdb user manually as per Example 2 
useradd -d /usr/local/var/lib/couchdb couchdb

### This is right
# change file ownership from root to couchdb user and adjust permissions
chown -R couchdb: /usr/local/var/{lib,log,run}/couchdb /usr/local/etc/couchdb
chmod 0770 /usr/local/var/{lib,log,run}/couchdb/
chmod 664 /usr/local/etc/couchdb/*.ini
chmod 775 /usr/local/etc/couchdb/*.d

# start couchdb
cd /etc/init.d
ln -s /usr/local/etc/init.d/couchdb couchdb
/etc/init.d/couchdb start
# Start couchdb on system start
update-rc.d couchdb defaults

# Verify couchdb is running
curl http://127.0.0.1:5984/
# {"couchdb":"Welcome","version":"1.0.1"}

###### OMG! Up and running!
# cd
# /etc/init.d/couchdb start
 * Starting database server couchdb
   ...done.
# update-rc.d couchdb defaults
 Adding system startup for /etc/init.d/couchdb ...
   /etc/rc0.d/K20couchdb -> ../init.d/couchdb
   /etc/rc1.d/K20couchdb -> ../init.d/couchdb
   /etc/rc6.d/K20couchdb -> ../init.d/couchdb
   /etc/rc2.d/S20couchdb -> ../init.d/couchdb
   /etc/rc3.d/S20couchdb -> ../init.d/couchdb
   /etc/rc4.d/S20couchdb -> ../init.d/couchdb
   /etc/rc5.d/S20couchdb -> ../init.d/couchdb
# curl http://127.0.0.1:5984/
{"couchdb":"Welcome","version":"1.0.2"}
# 
###### end OMG! Up and running!

# run test suite in firefox at http://127.0.0.1:5984/_utils/couch_tests.html?script/couch_tests.js
### This won't work at all, because Futon is only available at this point on the "local" machine 
### and we have a server and we don't want folks taking a nap on our futon!
### ... so see next section, on how to easily set up a secure tunnel to run futon locally on a dev machine
### ... and on how to set up remote access

Securely Admin CouchDB with an SSH Tunnel

See http://library.linode.com/databases/couchdb/ssh-tunnel

I have attached the perl script couchdb-tunnel.pl found on that page, edited the configurate found in the first two lines (lines 3 and 4 left at 5984 and 127.0.0.1 respectively), and run it on both Ubuntu desktop and Mac Pro, and obtained the following results:

Secure administration of CouchDB using an ssh pipe

Setting up remote access

In a manner analogous to MySql, by default CouchDB is set up for local access only, through 127.0.0.1 . You can change this just as easily, by editing the file /usr/local/etc/couchdb/local.ini and setting bind_address to 0.0.0.0

Now you can have apps going on with design deocuments, etc.

However, this will make your server insecure, and you will need to start using authentication and authorization for administration and editing. Check out the lower right-hand corner of Futon, where it says “Welcome to Admin Party! Everyone is admin. Fix this (link)” to get started.

So CouchDB is installed, now wot?

Couple of cool resources to get you started:

Installing Node.js

The following pages give good background on installing node.js and npm in various situations, including without root access and even without sudo access:

  • https://github.com/joyent/node/wiki/Installation
  • http://joyeur.com/2010/12/10/installing-node-and-npm/
    • https://gist.github.com/579814
  • http://brianstoner.com/blog/setting-up-nodejs-express-couchdb-dev-environment

In any case, I do want to install as root, making the install default simple and server wide, and maintainable. So I chose accordingly.

Dependencies for installing node.js

“Node.js itself has no external dependencies except common build tools as well as pythons for the build system itself. On OSX you must install XCode for this to work, and on Ubuntu you probably have to run: $ apt-get -y install build-essential” – Felix’s Node.js Beginner’s Guide http://nodeguide.com/beginner.html#installation

 

  • build-essentials
  • curl
  • openssl (libssl-dev)
  • python for node’s build tools
  • git

All set to go on these after installing CouchDB as per above steps. To install git:

### So the following is not necessary unless starting out fresh from this point.
# apt-get install build-essential curl libssl-dev
### Install latest git from source
# cd /opt
# apt-get remove git-core
# apt-get build-dep git-core
### Went to http://git-scm.com/download and got the link to the source for latest stable version
# wget http://kernel.org/pub/software/scm/git/git-1.7.5.2.tar.bz2
# tar xvjf git-1.7.5.2.tar.bz2
# cd git-1.7.5.2
# ./configure
# make
# make install
# cd
# git --version
git version 1.7.5.2

Installing node.js

Note: This is actually the most straightforward way, once the dependencies have been dealt with:

Go to http://nodejs.org/#download and grab the tarball (do this in /opt or wherevery you like):

$ wget http://nodejs.org/dist/node-v0.4.8.tar.gz
$  tar xvzf node-v0.4.8.tar.gz
$ cd $ cd node-v0.4.8
$ ./configure
$ sudo make install

(See https://github.com/joyent/node/wiki/Installation for installing locally, relative to your $HOME)

Installing latest stable version from git as root:

# cd /opt
# git clone git://github.com/joyent/node.git
# cd node
# git checkout origin/v0.4 # optional.  Note that master is unstable. Ignore 'detached HEAD'... 
           # Alternative (check out tag): $ git checkout v0.4.8
# ./configure
# make install
# which node
/usr/local/bin/node

Now test the node console:

 node
> var foo = {bar: 'baz'};
> console.log(foo);
{ bar: 'baz' }
> .help
> .exit

Edit: Very important to add checkout statement for the latest stable version, instead of just Master unless you are experimenting. To find out which is the current most stable version, go to http://nodejs.org/#download and to see. At the time of writing, it was node-v0.4.8.tar.gz, which can be installed by unpacking and running configure and make install as detailed above.

Installing npm (Node.js package manager)

# curl http://npmjs.org/install.sh | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
101  3874  101  3874    0     0  41540      0 --:--:-- --:--:-- --:--:-- 65661
fetching: http://registry.npmjs.org/npm/-/npm-1.0.6.tgz
0.5.0-pre
1.0.6
prefix=/usr/local

This script will find and eliminate any shims, symbolic
links, and other cruft that was installed by npm 0.x.

Is this OK? enter 'yes' or 'no' 
yes

All clean!
! [ -d .git ] || git submodule update --init
node cli.js cache clean
node cli.js rm npm -g -f --loglevel error
node cli.js install -g -f
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm.js
node-uuid@1.1.0 /usr/local/lib/node_modules/npm/node_modules/node-uuid
npm@1.0.6 /usr/local/lib/node_modules/npm
It worked
# which npm
/usr/local/bin/npm

So node.js is installed, now wot?

  • http://ofps.oreilly.com/titles/9781449398583/
  • http://camp.nodejs.org/
    • videos: http://camp.nodejs.org/videos/
    • resources: http://camp.nodejs.org/resources.html
  • http://www.nodebeginner.org/
  • http://axonflux.com/nodejs-beginner-links
  • http://nodetuts.com/
  • Interesting profile on importance of Node.js: http://www.theregister.co.uk/2011/03/01/the_rise_and_rise_of_node_dot_js/

Installing Express

According to instructions at https://github.com/visionmedia/express I did the following in order to access the express executable globally:

# npm install -g express
/usr/local/bin/express -> /usr/local/lib/node_modules/express/bin/express
connect@1.4.1 /usr/local/lib/node_modules/express/node_modules/connect
mime@1.2.2 /usr/local/lib/node_modules/express/node_modules/mime
qs@0.1.0 /usr/local/lib/node_modules/express/node_modules/qs
express@2.3.7 /usr/local/lib/node_modules/express

Then, as a dev user, I create a first create a skeleton

$ express testapp
   create : testapp
   create : testapp/app.js
   create : testapp/views
   create : testapp/views/layout.jade
   create : testapp/views/index.jade
   create : testapp/public/stylesheets
   create : testapp/public/stylesheets/style.css
   create : testapp/public/images
   create : testapp/public/javascripts
   create : testapp/logs
   create : testapp/pids
   create : testapp/test
   create : testapp/test/app.test.js

Then I cd into it and install express and jade dependencies

$ cd testapp
~/testapp$ npm install express jade
jade@0.11.0 ./node_modules/jade
mime@1.2.2 ./node_modules/express/node_modules/mime
connect@1.4.1 ./node_modules/express/node_modules/connect
qs@0.1.0 ./node_modules/express/node_modules/qs
express@2.3.7 ./node_modules/express

Then I can execute the test app:

$ node app.js 
Express server listening on port 3000

And point my browser at http://my-vps-server.com:3000/

 

Express

Welcome to Express

 

Installing additional components

[tbd]