This is a first longish article in what is to be a longish series of articles on how working stiffs can have their very own Drupal development server, with all the tools they need to develop, test and deploy and monitor their Drupal projects, for free. As such it is techie but only enough to get the job done.
Drupal site monitoring (site up? new versions or releases need to be installed, especially security releases? etc., etc.) will be integrated into the Project Flow and Tracker Drupal development framework installation profile (the heart of the development server) via Nagios, so that when you start a project, the associated test site can be monitored as you work, whether local or remote.
Sign up for a free account and test out Project Flow & Tracker (still in alpha but perfectly usable, check out the podcast on how to set up a project): http://projectflowandtracker.com/
See Project Flow & Tracker DrupalConDC 2009 session proposal here: http://dc2009.drupalcon.org/session/project-flow-and-tracker-business-objects-and-user-stories-test-driven-drupal-based-website-
Track Project Flow & Tracker development progress here: http://groups.drupal.org/node/16850
Contents of this article:
- Readying the server
- Publishing the server on the Internet with dynamic DNS
- Installing Nagios
- First steps with Nagios
- Next steps in future articles
Readying the server
A few weeks ago, just after I transferred all my work over to my brand new Dell Inspiron 1420n Ubuntu notebook, my home desktop machine, a wonderful AMD64 creature, blew a power supply. Since it has been sitting there doing nothing, I decided to get it fixed and convert it into a home office network and web server. And then I decided to turn it into a Drupal Development Server, housing the Project Flow and Tracker system for all the world to see as it grows from baby boy to handsome young application.
Installation
I downloaded the Ubuntu 8.10 Server edition (64bit version, in my case) from the Ubuntu download page (choosing a local mirror): http://www.ubuntu.com/getubuntu/download , burned it into a CD and revved it up on mentor.
It was really easy to install (if you have any doubts, see http://www.howtoforge.com/perfect-server-ubuntu-8.10 ). Also, when I got to the Software selection screen, I just checked off the automatic install of:
- LAMP server (provide root password during install of MySql)
- Mail server (postfix!)
- OpenSSH server
- Tomcat Java Server (nostalgia)
- Virtual machine host (got to look into that later…)
After a few more steps I was told to remove the CD and reboot into the new system.
Configuration after the initial reboot
I did not enable any root password (just a user, then I can work with “sudo -s” or “sudo su” to get a root shell whenever I want).
Static IP
What is important is to provide a static IP for your server, especially if you need to do port forwarding from behind a router. Whether or not that means anything to you, just give the server a static IP. To do so (covered in HowToForge article, but repeated here for the sake of redundancy), edit the file /etc/network/interfaces, and where you see the line:
iface eth0 inet dhcp
put in something like the following (random values):
# The primary network interface auto eth0 iface eth0 inet static address 192.168.1.77 netmask 255.255.255.0 network 192.168.0.0 broadcast 192.168.0.255 gateway 192.168.1.1
Here the gateway is a Linksys router, configured to hand out IP’s via DHCP starting with 192.168.1.100 — make sure the static IP you give the server is below or way above any such threshold you establish, to avoid any possible conflicts. Follow the instructions in the above HowTo (which you should get acquainted with if you are going to use the Intrepid server) in terms of the hostname and editing of /etc/hosts, whose first two lines should look something like this:
127.0.0.1 localhost 192.168.1.77 mentor
Either reboot or restart the network with the following command:
# /etc/init.d/networking restart
Updating the system
First edit /etc/apt/sources.list. The CD should no longer be a source, and you need to enable universe and multiverse repositories. In my case the installation CD was already commented out from the file, but I did make sure that both the universe and multiverse repositories were enabled (see HowTo).
Then I updated the sources and updated the system:
# apt-get update
# apt-get upgrade
Make bash the default shell
I didn’t want to delete /bin/sh, just in case someone wants it sometime, so I did:
# mv /bin/sh /bin/sh.old # ln -s /bin/bash /bin/sh
Apache configuration
In order for clean URLs to work in Drupal, you need to enable .htaccess files and install mod rewrite. First I edited /etc/apache2/sites-enabled/000-default.
Before (lines 10-18):
<Directory /var/www/> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all ...
After:
<Directory /var/www/> Options Indexes FollowSymLinks MultiViews AllowOverride All ...
Then I executed the following in order to enable mod_rewrite and restart apache server:
# a2enmod rewrite Module rewrite installed; run /etc/init.d/apache2 force-reload to enable.
# apache2ctl restart
The apache server restarted ok, but I got this pesky error:
apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
I edited /etc/apache2/apache2.conf and added a ServerName directive just under the ServerRoot directive:
ServerRoot "/etc/apache2" ServerName hostname
Where hostname should be whatever you get when you execute “hostname”. You should get an error message free apache server restart now.
Install cvs and subversion
# apt-get install cvs
# apt-get install subversion
Install phpmyadmin
# apt-get phpmyadmin
When you point a browser in another computer on the local network to http://hostname/phpmyadmin (after specifying hostname in your local /etc/hosts file, of course), you login as root with the password you specified during MySql installation.
An Intrepid SSH problemita
Looking at my syslog while debugging ddclient (see below) I saw the following:
Nov 14 06:40:01 my-server console-kit-daemon[16105]: CRITICAL: cannot initialize libpolkit Nov 14 06:40:01 my-server /USR/SBIN/CRON[16168]: (root) CMD ([ -x /usr/sbin/update-motd ] && /usr/sbin/update-motd 2>/dev/null) Nov 14 06:40:02 my-server CRON[16102]: Sigfile not found
This fixes it (some people have reported dire consequences from this problem, like having to reboot to use SSH login after a 24-hour period):
# sudo apt-get install policykit
Getting the new version of pftracker as a vanilla Drupal 6 install up and running on a virtual server
This is what I did:
- Made a developer user on the system
- Logged in as user “developer” I created a “www” directory under /home/developer
- I changed directory to /home/developer and did:
- # cvs -z6 -d:pserver:anonymous:anonymous@cvs.drupal.org:/cvs/drupal co -d pftracker -r DRUPAL-6-6 drupal
- This downloaded a fresh Drupal 6.6 install straight from the repository. Now, when I need to upgrade, I can just do something like:
- cvs update -dPr DRUPAL-6-7
- I created a new MySql database and user with the same name on the server using PhpMyAdmin
- I added the following lines (notice “*:80” with the standard Intrepid install) at the foot of /etc/apache2/sites-enabled/000-default :
-
<VirtualHost *:80> ServerName pftracker.hostname DocumentRoot /home/developer/www/pftracker <Directory "/home/developer/www/pftracker"> Options Indexes MultiViews FollowSymLinks AllowOverride All Order deny,allow Deny from all allow from all </Directory> </VirtualHost>
-
- Yet another apache server restart
- On my laptop on the same LAN, I added pftracker.hostname to my local /etc/hosts file pointing it to the same IP as hostname)
- I pointed my browser at http://pftracker.hostname, and birthed Project Flow & Tracker on Drupal 6, specifying the database I had created.
Now that I can see it, I want everyone else to! So…
Publishing the server on the Internet with dynamic DNS from behind the Linksys router
I did it both ways: DynDNS and ZoneEdit. The former uses one of DynDNS’ hostnames in the free version, while with ZoneEdit I am going to use a domain of my own. Further, DynDNS is further simplified
The DynDNS way (Lynksys router refresh)
I figure I can trust any geek who has read this far into the article, by letting them see that they can see the new-born pftracker at http://pftracker.mentor.dnsdojo.org/ . How was this wondrous feat achieved?
First I went to the wonderful folks at http://www.dyndns.com/ and signed up for an account. After logging in, I hit the My Services option on the left sidebar, and in the Host services section clicked on Add Hostname.
I typed in “mentor” (hostname) and chose from among the colorful domains offered (example: mentor.isa-geek.com). I chose dnsdojo.org, being partial to the word “dojo”.
Important: I selected the Wildcard checkbox! That way, all subdomains set up as virtual hosts will also work! For example, http://pftracker.mentor.dnsdojo.org/
I clicked on the Use auto detected IP address xx.xx.xx.xx (DynDNS automatically detects your effective IP). Then I clicked on the Create Host button. From that point in time, anyone in the whole wide world pointing their browsers at pftracker.mentor.dnsdojo.org will… get to my router.
Actually, I had two tasks before me:
- Tell server mentor where the document root of the URL is http://pftracker.mentor.dnsdojo.org/
- Tell the router to forward port 80 requests to server mentor
The first task involved configuring the Virtual server to catch requests at that URL and send them to the appropriate DocumentRoot. So I changed the virtual host configuration to the following:
<VirtualHost *:80> ServerName pftracker.mentor.dnsdojo.org DocumentRoot /home/developer/www/pftracker <Directory "/home/developer/www/pftracker"> Options Indexes MultiViews FollowSymLinks AllowOverride All Order deny,allow Deny from all allow from all </Directory> </VirtualHost>
Then, I logged into my Wireless-G Broadband Linksys router through its internal LAN web interface and clicked on the Applications and Gaming menu item (it’s all a game, don’t you know…). There, I specified a port range of 80 – 80, a protocol of TCP, and the server IP address (that’s why it has to have a static IP address!) and selected the Enable checkbox, then clicked on the Save Settings button. Now, any http request coming into the router will go to the server. Whatever router you are using they all have some way of configuring NAT or port forwarding in a similar way.
Then, from the main page of the Linksys admin page, I hit the DDNS menu tab. I selected DynDNS from the two services offered, entered my DynDNS User Name and Password, entered the hostname (mentor.dnsdojo.org) and hit the Save settings button.
I was in business: “Update Success”. Now, when my ISP changes my IP, my fearless router will let DynDNS know. Cool.
But what I really want is to use one of my own domains, make it a bit more professional like. But to manage a “zone” you have to pay out good money. Enter stage left ZoneEdit and ddclient (since ZoneEdit does not seem to be one of the services offered by Cisco/Linksys)…
The ZoneEdit way (ddclient refresh)
ZoneEdit is a great service, offering a myriad of services, most of them free unless you have a huge domain to manage. See https://www.zoneedit.com/ for a list of all their services. “Full free dynamic DNS support, allowing cable modem, dsl, and dial up users to run web sites on your home pc.” And, I might add, doing it with any one of five of your own domains (zones) for free, as in free beer.
Since it is not extremely obvioius how to get this working, I will let you know how I did it.
We need to accomplish two tasks here:
- Setting up a Zoneedit account and managed zone (domain).
- Installing software to update that account if our IPS changes our IP (Linksys router doesn’t include that service).
I went to https://www.zoneedit.com/ (notice the https, folks) and hit the Free Sign Up link. Enter your personal details and select the Free Trial account (has been giving you 5 free zones to manage for years) and sign up. Login to your account and click on the Add Zones link. Enter myhotnewdomainname.com (be careful, you can’t erase these, and if you do it without thinking you will use up your five free zones fast!!!) and hit the Add Zone button.
Edit the zone, and add an A record (an IP address). Put in myhotnewdomainname.com and enter what you think your current IP address to be (don’t worry, that will be done automatically for you later by ddclient — see below). You probably want to have that IP used for both www.myhotnewdomainname.com and myhotnewdomainname.com as well.
Now, this can be a domain name you have registered with zoneedit.com itself, or one with another registrar. Mine is registered with nic.ar, the Argentine internet registrar, which offers free domains. So Zoneedit told me to point my nameservers to two zoneedit nameservers. And that in 72 hours the domain will actually be working with zoneedit. Great. Did that. It works.
Now we need some way to dynamically inform zoneedit of changes in the server’s IP. Enter ddclient. Here’s how I did it:
# apt-get install ddclient
I then edited /etc/default/ddclient and set run_daemon to “true” and left the daemon_interval at 300 seconds.
To see what was going on with the daemon, I ran:
# /etc/init.d/ddclient status
It wasn’t running, so I started the daemon with:
# /etc/init.d/ddclient start
But I stopped it immediately because nothing is going to happen until you edit the configuration file, so I stopped it with:
# /etc/init.d/ddclient stop
and edited /etc/ddclient.conf as follows (it works)
# Configuration file for ddclient generated by debconf # # /etc/ddclient.conf pid=/var/run/ddclient.pid protocol=zoneedit1 use=web, web=dyndns server=dynamic.zoneedit.com login=username password='password' wildcard=YES encompass.com.ar, www.encompass.com.ar, pftracker.encompass.com.ar, pftlegacy.encompass.com.ar
Notice that the last line is a comma separated list of URLs that should be associated with the server’s dynamic IP
(Of course, when it starts working, all will use the /var/www document root until I configure a Virtual Server with apache2).
Next: validate the syntax, run it once in forced mode, start it up as a daemon, check the log.
To validate the syntax, I ran:
# ddclient -query use=if, if=eth0 address is 192.168.1.77 use=if, if=lo address is 127.0.0.1 use=if, if=vnet0 address is 192.168.129.1 use=web, web=dnspark address is 201.231.222.22 use=web, web=dyndns address is 201.231.222.22 #
The values are ficticious, but you should see something like that.
To run it once forcing the values (so I can go to my Zoneedit account and see if they are actually done, I do:
# root@mentor:~# ddclient -force SUCCESS: updating encompass.com.ar: IP address set to 201.231.222.22 (201: No records need updating.) SUCCESS: updating pftlegacy.encompass.com.ar: IP address set to 201.231.222.22 (201: No records need updating.) SUCCESS: updating pftracker.encompass.com.ar: IP address set to 201.231.222.22 (201: No records need updating.) SUCCESS: updating www.encompass.com.ar: IP address set to 201.231.222.22 (201: No records need updating.)
The success shows that when you go to your account on ZoneEdit, each entry will have been added if it wasn’t there before, and the IP values entered; or, as in this case, a return value indicates that no values needed changing (since I had run this earlier).
To allow ddclient to continue life as a daemon, I did the following:
# /etc/init.d/ddclient start
Check out your ddclient history for the day:
root@hostname:/var/log# grep ddclient /var/log/syslog.0 Nov 22 11:18:32 hostname ddclient[9689]: SUCCESS: updating encompass.com.ar: IP address set to 201.231.249.92 (200: Update succeeded.) Nov 22 11:24:04 hostname ddclient[9689]: WARNING: caught SIGTERM; exiting Nov 22 11:24:51 hostname ddclient[9816]: WARNING: caught SIGTERM; exiting Nov 22 11:24:56 hostname ddclient[9835]: WARNING: skipping host: : 'login=' is an invalid login. Nov 22 11:24:56 hostname ddclient[9835]: WARNING: skipping host: web=dyndns: 'login=' is an invalid login. Nov 22 11:24:56 hostname ddclient[9838]: WARNING: skipping host: : 'login=' is an invalid login. Nov 22 11:24:56 hostname ddclient[9838]: WARNING: skipping host: web=dyndns: 'login=' is an invalid login. Nov 22 11:25:11 hostname ddclient[9838]: WARNING: caught SIGTERM; exiting Nov 22 11:25:14 hostname ddclient[9850]: WARNING: skipping host: : 'login=' is an invalid login. Nov 22 11:25:14 hostname ddclient[9850]: WARNING: skipping host: web=dyndns: 'login=' is an invalid login. Nov 22 11:25:14 hostname ddclient[9853]: WARNING: skipping host: : 'login=' is an invalid login. Nov 22 11:25:14 hostname ddclient[9853]: WARNING: skipping host: web=dyndns: 'login=' is an invalid login. Nov 22 11:25:18 hostname ddclient[9853]: WARNING: caught SIGTERM; exiting Nov 22 20:12:08 hostname ddclient[9867]: SUCCESS: updating pftracker.encompass.com.ar: IP address set to 201.231.222.22 (200: Update succeeded.) Nov 22 20:12:47 hostname ddclient[9867]: WARNING: caught SIGTERM; exiting
So ddclient is all set up. When you reboot the server, and login, you should see it’s alive:
# ps -ef | grep ddclient
root 4080 1 0 13:51 ? 00:00:00 ddclient – sleeping for 170 seconds
murgetroyd 4687 4664 0 13:53 pts/0 00:00:00 grep ddclient
victorkane@mentor:~$
Now we need to get network monitoring going, on the basis of a good solid piece of widely used open source software with an API, so that it can be integrated into the Project Flow & Tracker development system running on our server.
Installing Nagios
Nagios has in common with Drupal architecture that it has a very basic core, with most useful functionality added via plugins. Full documentation for compiling Nagios from source and installing it on a wide variety of systems is available at http://nagios.sourceforge.net/docs/3_0/quickstart.html . On the Intrepid Ibex 8.10 server I did the following:
# apt-get install nagios3
I then went to /etc/nagios3 and configured Nagios authentication by creating an htpasswd file for user nagiosadmin:
# htpasswd -c htpasswd.users nagiosadmin
and supplied a strong password.
I pointed my browser at http://hostname/nagios3 (notice the 3, given the ubuntu package) and after logging in with the password I specified earlier was greeted at the home page, and was able to see the status of two default servers, the server itself and the Linksys router, and, most importantly, was able to settle down with a freshly brewed Argentine mate green tea and read the extensive documentation (well, the Quickstart anyway…).
Clicking on the Host Detail sidebar menu option, I could see my linksys router (identified as gateway) and localhost specified, with lastcheck timestamps, durations (i.e. up since…) as well as status information ( PING OK – Packet loss = 0%, RTA = 0.07 ms ) for each entry.
First steps with Nagios
Basically I added a host and an http service for it, so I could be informed by mail if that Drupal project stops responding.
First of all I needed to know where to specify hosts and services, so viewed the file /etc/nagios3/nagios.cfg and near the end saw the following line:
# Debian uses by default a configuration directory where nagios3-common, # other packages and the local admin can dump or link configuration # files into. cfg_dir=/etc/nagios3/conf.d
Cool, this is what the Nagios documentation meant when it said, in the section entitled “Monitoring Publicly Available Services”:
“You can place host definitions in any object configuration file specified by a cfg_file directive or placed in a directory specified by a cfg_dir directive.”
That means I can stick config files into the /etc/nagios3/conf.d directory! Just like the installation process created one for the Linksys router, the file /etc/nagios3/conf.d/host-gateway_nagios3.cfg . To add textwerks.com (fictional name), which hosts one of my Drupal projects, I cunningly copied this file to another, called host-textwerks_nagios3.cfg and edited it:
define host {
host_name textwerks
alias textwerks.com
address textwerks.com
use generic-host
}
I immediately did a sanity check for the configuration by running:
# nagios3 -v /etc/nagios3/nagios.cfg Nagios 3.0.2 Copyright (c) 1999-2008 Ethan Galstad (http://www.nagios.org) Last Modified: 05-19-2008 License: GPL Reading configuration data... Running pre-flight check on configuration data... Checking services... Checked 7 services. Checking hosts... Warning: Host 'textwerks' has no services associated with it! Checked 3 hosts. Checking host groups... Checked 5 host groups. Checking service groups... Checked 0 service groups. Checking contacts... Checked 1 contacts. Checking contact groups... Checked 1 contact groups. Checking service escalations... Checked 0 service escalations. Checking service dependencies... Checked 0 service dependencies. Checking host escalations... Checked 0 host escalations. Checking host dependencies... Checked 0 host dependencies. Checking commands... Checked 142 commands. Checking time periods... Checked 4 time periods. Checking for circular paths between hosts... Checking for circular host and service dependencies... Checking global event handlers... Checking obsessive compulsive processor commands... Checking misc settings... Total Warnings: 1 Total Errors: 0 Things look okay - No serious problems were detected during the pre-flight check
Well, I had textwerks set up OK, just hadn’t defined any services. Actually didn’t have to, since all hosts would be pinged by default I would think. To enable that I needed to restart Nagios:
# /etc/init.d/nagios3 reload * Reloading nagios3 monitoring daemon configuration files nagios3 [ OK ]
Or, stop and start:
# /etc/init.d/nagios3 stop * Stopping nagios3 monitoring daemon nagios3 [ OK ] # /etc/init.d/nagios3 start * Starting nagios3 monitoring daemon nagios3 [ OK ] #
Then, pointing my browser once again to http://hostname/nagios3/ and then choosing Host Detail, our newly specified host is listed, with its corresponding data.
A very good source of information, apart from the great Nagios docs themselves, is: Nagios 3 Enterprise Network Monitoring: Including Plug-Ins and Hardware Devices
Next steps in future articles
In the next few articles we will add a lot of functionalilty to Nagios, learn how to access reports on the command line and through the API, and then integrate Nagios with the Project Flow & Tracker application running on the same Drupal Development Server.