Administrator Guide to oolong.uta.fi

Chienting Weng
Hypermedia Laboratory
University of Tampere

Version 0.1 --- 28.03.2008

Chapter 1: Introduction

This document is for the administrator of oolong, a Dell PowerEdge 1950 (abbr is PE1950) rack server that host Hypermedia Lab Game Research Group's website and web services. The document describes the hardware specifications, operating systems, virtualization tools, services, and related configurations running on oolong.

Chapter 2: Server Specifications

This chapter includes the hardware specifications and system architectures of oolong.

2.1 Hardware Specifications

All hardwares are originally come with PE1950, there are no extra parts.

Fig1. Hardware Specification of PE1950
Item Specification Quantity
Model Dell PowerEdge 1950 1U rack server 1
CPU Intel Xeon Quad Core 1
Hard Disk Drive 143GB SAS HDD 2
RAM 2GB 2
Power 2
RAID Controller Dell PERC 5/i SAS/SATA/SCSI RAID Controller (LSI MegaRAID) 1
Network Broadcom NetXtreme II Gigabit Ethernet 2
Support 3 Year, 4 hour 24/7 onsite sSilver upport N/A
Operating System None None

2.2 System Architecture

The system uses Xen virtualization software to run three virtual systems. Xen is an open source virtualization solution which supports a wide range of virtual operating systems including Windows, Linux, Solaris, and various versions of the BSD operating systems. It is theorically possible to run all mentioned operating systems on Xen, but due to the host OS and hardware, it is simply more difficult to get some operating systems working well than others, for example, I could not run FreeBSD well on a Debian-based Xen box. But there are more success stories about running Windows and Linux on Xen.

The base operating system (called Dom0 in Xen) is Debian GNU/Linux 4.0 (a.k.a Debian Etch), the virtual systems (called DomU in Xen) running on it are two Ubuntu 6.06 LTS and one Debian Etch. The respective operating systems are listed in the table below:

Fig2. System architecture
Dom0/DomU Operating System
Dom0 Debian Etch
DomU Ubuntu 6.06 LTS
DomU Debian Etch
DomU Ubuntu 6.06 LTS

2.2.1 Network Interfaces, IPs and hostnames

There are six static IPs available for use, two are for the physical ethernet network interface on Dom0 with specified Mac address, four are for DomU virtual machines, listed as below:

Note: because the server will be moved to university's computer center, all the IP addresses will change after moving.

Fig3. Network Interface Information
NIC/DomU IP Hostname
NIC (mac: 00:19:B9:EB:BE:85) 153.1.17.22 oolong.uta.fi
NIC (mac: 00:19:B9:EB:BE:87) 153.1.17.26 oolong1.uta.fi
DomU 153.1.17.86 oolong2.uta.fi
DomU 153.1.17.87 oolong3.uta.fi
DomU 153.1.17.88 oolong4.uta.fi
DomU 153.1.17.89 oolong5.uta.fi

Chapter 3: System Installation

This chapter describes the system-level configurations and how to install required operating systems on oolong.

3.1 RAID System

oolong uses RAID 1 (one-to-one mirror) backup, so the actually available HDD space is 143 GB, the other HDD is for mirroring. To modify the RAID configuration, press Ctrl+R to enter the PERC 5/i controller utility at booting.

3.2 Debian Etch Installation

To install Debian Etch, download the CD/DVD image from http://ftp.funet.fi/pub/linux/mirrors/debian-cdimage/. No special tweaks are required during the installation process, network card and RAID controller are automatically detected and activated. But the network driver has to be upgraded and the network configuration has to be modified after installation.

3.2.1 Network Interface Card Configuration

Dell PE1950 present their onboard network cards as Gb1 and Gb2 in BIOS, but these map to eth1 and eth0 respectively ("backwards") under Linux. This issue is fixed in kernel 2.6.19-rc3 and higher. But the current kernel of oolong is kernel 2.6.18-5, so manually changing the network card configurations is necessary. In our case, the default dhcp configuration will assign IP addresses and Hostnames to each network card as following:

Fig4. Ethernet Card Configurations
Mac address Card name IP Hostname
00:19:B9:EB:BE:85 eth1 153.1.17.26 oolong.uta.fi
00:19:B9:EB:BE:87 eth0 153.1.17.22 oolong1.uta.fi

This is different from our expectation: eth0 as oolong.uta.fi and eth1 as oolong1.uta.fi. The solution is to switch the network mac address, make 00:19:B9:EB:BE:85 become eth0's mac address and 00:19:B9:EB:BE:87 become eth1s'. To change mac address information, edit /etc/udev/rules.d/z25_persistent-net.rules. The modified file is as below:

# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.
# MAC addresses must be written in lowercase.

# PCI device 0x14e4:0x164c (bnx2)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:19:b9:eb:be:87", NAME="eth1"

# PCI device 0x14e4:0x164c (bnx2)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:19:b9:eb:be:85", NAME="eth0"
	

In this way, we can guarantee that dhcp will assign correct IP to each ethernet network card. /etc/udev/rules.d/z25_persistent-net.rules is automatically generated at first time startup, after that, it will be modified if there are new interfaces hardware added to the system, but the existing content will remain the same. So unless you reinstall the system or manually modify the existing content, otherwise, the configuration file will not be changed. More information about /etc/udev/rules.d/z25_persistent-net.rules can be found at Debian Wiki udev.

3.2.2 Add Ethernet Network Card

Only one network card is detected and activated at installation, so we have to manually add and configure another ethernet card. For a detail document on how to configure network on Debian, refer to Debian Wiki - NetworkConfiguration. The network interface configuration file is /etc/network/interfaces, the current content is as following:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
allow-hotplug eth0
# iface eth0 inet dhcp
iface eth0 inet static
address 153.1.17.22
netmask 255.255.255.0
gateway 153.1.17.5

auto eth1
allow-hotplug eth1
# iface eth1 inet dhcp
iface eth1 inet static
address 153.1.17.26
netmask 255.255.255.0
gateway 153.1.17.5

up /sbin/iptables-restore /etc/network/iptable
    

Although as mentioned in 3.2.1 that dhcp works, I personally perfer specifying network configuration information manually, just a double check. Besides the default eth0, eth1 and its configurations are added to the file. The last line up /sbin/iptables-restore /etc/network/iptable activates the firewall. After changing /etc/network/interfaces, it is required to restart the network to make the new configuration working. To relaunch the network, run /etc/init.d/networking restart with root permission.

3.2.3 Upgrade Network Driver

The original Broadcom NetXtreme II network card driver (bnx2 v1.4.44) that comes with Debian Etch can not work properly on Xen, this issue is solved in version 1.5.10c, so the network card driver has to be upgraded to at least or higher than 1.5.10c. To download the latest driver, visit http://www.broadcom.com/support/ethernet_nic/netxtremeii.php, or there is a copy of bnx2 1.5.10c driver at //arnika/laitos$/hyper/Projektit/Gamelab/Server_documents/PE1950. Refer to the xensource mailing list for the detail instructions. The complete email content is copied as below:

I have a new x3655 IBM and whenever the /etc/xen/scripts/network-bridge
script starts, the ethernet would no longer work.  This post isn't about
the cause, but the fix.  I did a lot of tcpdumps and Googling, but I'll
spare you that.  All they do is prove that yes, there is an issue. :)

I tried Debian 4.0 i386/amd64 and Ubuntu 7.04 Server i386/amd64 and they
all exhibit the problem.

Upgrading the bnx2 driver is the key.  I came to find bnx2 is included
in the initrd.  Debian comes with 1.4.44 and Ubuntu 1.4.45 of the
driver.  As of this writing 1.5.10c is the latest from Broadcom and is
what I ended up using.
	
Summary:
* Install kernel source, make config/prepare the distro's config file
* Download bnx2 source, compile and install
* Update initrd

Here are step by step instructions for clean install of Debian 4.0 amd64:

aptitude install linux-source unzip
cd /usr/src
tar jxf linux-source-2.6.18.tar.bz2
ln -s linux-source-2.6.18 linux
cd linux
cp /boot/config-2.6.18-5-xen-amd64 ./.config
make oldconfig ( through all the prompts to take defaults)
make prepare
cd /tmp
wget http://www.broadcom.com/docs/driver_download/NXII/linux-1.5.10c.zip
unzip linux-1.5.10c.zip
cp Server/Linux/Driver/bnx2-1.5.10c.tar.gz /usr/src
cd /usr/src
tar zxf bnx2-1.5.10c.tar.gz
cd bnx2-1.5.10c/src
make
make install
mv /boot/initrd.img-2.6.18-5-xen-amd64 \
/boot/initrd.img-2.6.18-5-xen-amd64.orig
update-initramfs -k `uname -r` -c
reboot

Verify:
zcat /boot/initrd.img-2.6.18-5-xen-amd64.orig|strings|grep "bnx2 v"
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.4.44 (August 10, 2006)

zcat /boot/initrd.img-2.6.18-5-xen-amd64|strings|grep "bnx2 v"
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.5.10c (May 4, 2007)
	
Other keywords: Dell DELL

-- 
Kris S. Amundson
Founder, CIO                   GPG Key: D6D39F2C
OpenSourcery, LLC.             http://www.opensourcery.com/
	

3.3 Xen Installation

To install Xen, follow the The Perfect Xen 3.1.0 Setup For Debian Etch (i386) on Howtoforge. My experience with installing Xen from the binary package and from source all went smoothly, so the installation should not be a problem. The major difference between binary package and source is the ability to compile kernel for the virtual system. If you prefer customizing kernels for different virtual systems, you should install Xen from source. That is what I did to oolong. In The Perfect Xen 3.1.0 Setup For Debian Etch (i386), Page 5 and Page 6 decribe how to install Xen 3.1 from source step-by-step. Be sure to double check the driver version of the network card after Xen is installed, it must be higher than bnx2 1.5.10c.

If you plan to compile a new kernel for virtual system, besides enabling quota and iptables as modules in the kernel configuration menu (as mentioned in Page 5), you also have to enable SMB and CIFS network file systems support, otherwise, the system can not connect to our network hard disk drive. In the kernel configuration menu, you can find SMB and CIFS options at:

File Systems ---> 
Network File Systems ---> 
[M]SMB
[M]CIFS
    

Chapter 4: Xen Configuration

All Xen-related configuration and scripting files are placed in /etc/xen, xend-config.sxp is the core Xend configuration file.

4.1 Base system (Dom) Configurations

As a host of all other virtual machines, Dom0 itself does not run any particular services, only OpenSSL server so we can ssh to it. We need to configure the network so that Dom0 can provide network for all DomU and keep itself safe from unknown connections at the same time.

A basic introduction to Xen network can be found at http://wiki.xensource.com/xenwiki/XenNetworking.

4.1.1 Network Configuration

Xen provides two types of networking: bridging and routing, oolong uses bridging. For security reason, Dom0 does not share ethernet network card with DomUs. There are two ethernet network cards on oolong, eth0 is for Dom0, eth1 is for DomUs. The figure is illustrated as following:

The network configuration of oolong

To achieve above configuration, we do not use the default xend network configuration in xend-config.sxp. Instead, we create our own network bridge script and add it to xend-config.sxp. Open /etc/xen/xend-config.sxp, find the line (network-script oolong-network), oolong-network is the network bridge script created for our needs, you can replace this script with any other scripts you create for oolong.

All xen-related scripts are placed in /etc/xen/scripts, in /etc/xen/scripts, there is a file called oolong-network, the content is very simple:

dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth1 bridge=xenbr1
    

It merely calls the network-bridge script with vifnum=0 netdev=eth1 bridge=xenbr1 as parameters, then network-bridge will do the magic for us.

With those parameters, we specify eth1 as the network interface for DomU, and specify the name of bridge as xenbr1. A detailed example article can be found at http://www.debian-administration.org/articles/470.

4.1.2 Firewall Configuration

Besides the firewall settings in the university firewall, there are also firewall running on Dom0. The policy is: Dom0 is only accessible via eth0 (153.1.17.22/oolong.uta.fi), access to Dom0 via eth1 (153.1.17.26/oolong1.uta.fi) is not allowed. It is valid connecting to DomUs from Dom0, but connecting to Dom0 from DomUs is not allowed.

By now, only 153.1.17.87/oolong3.uta.fi is open for external http connection requests (port 80 and 443), other IPs are not open to the network outside our university, so they are only accessible from within the university network.

The firewall tool running on Dom0 is iptables, iptables is a firewall tool and is also a command to manage iptables rules in Linux. Use command man iptables to learn how to use iptables. After changing iptables rules, you can save the rules by using iptables-save. The iptables rules will be saved to /etc/network/iptable, and they will be automatically loaded when boot, so that you don't have to set up iptables rules every time. If you plan to change iptables rules, do not change /etc/network/iptable directly, instead, you should use iptables command, after you are sure the iptables rules work as expected, use iptables-save to save the rules. Before saving new iptables rules to /etc/network/iptable, you should back up the current one. The current iptables rules on oolong is as below:

# Generated by iptables-save v1.3.6 on Wed Jan 23 17:33:16 2008
*nat
:PREROUTING ACCEPT [2660:485548]
:POSTROUTING ACCEPT [1166:193315]
:OUTPUT ACCEPT [57:3666]
COMMIT
# Completed on Wed Jan 23 17:33:16 2008
# Generated by iptables-save v1.3.6 on Wed Jan 23 17:33:16 2008
*filter
:INPUT DROP [821:130868]
:FORWARD ACCEPT [72814:100421356]
:OUTPUT ACCEPT [613:72279]
-A INPUT -d 153.1.17.22 -i eth0 -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -i eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT 
COMMIT
# Completed on Wed Jan 23 17:33:16 2008
	

What those rules do are:

The iptables rules are automatically applied through /etc/network/interfaces at booting. To stop this, remove or comment /sbin/iptables-restore /etc/network/iptable in /etc/network/interfaces.

4.2 Virtual system (DomU) Configurations

xen-tool is used to create DomU images, a global configuration file (xen-tools.conf) is places in /etc/xen-tools/. Values can be set in this file so they don't need to be specified everytime upon the command line. The command to create a DomU image is xen-create-image, take oolong3.uta.fi for example, the command is:

xen-create-image --hostname=oolong3.uta.fi --ip=153.1.17.87 --dist=etch --mirror=http://ftp.fi.debian.org/debian/ --size=30Gb

Other parameters required for creating DomU images are specified in /etc/xen-tools/xen-tools.conf. You can also overwrite the values by specifying them upon the command line. xen-create-image creates the DomU image and configuration file for every created DomU image.

The configuration file of every DomU image are stored at /etc/xen/*.cfg. To make the network working, one line has to be modified for every .cfg file. In each .cfg file, there is a block like:

#
#  Networking
#
vif  = [ 'ip=,bridge=xenbr0' ]
    

Make sure the IP address is correct, then change the bridge=xenbr0 to bridge=xenbr1. In our configurations, the name of the bridge is xenbr1 not xenbr0.

This table lists the major configuration parameters of every DomU:

Fig5. Virtual System (DomU) Configurations
hostname ip dist mirror size memory swap
oolong2.uta.fi 153.1.17.86 dapper http://gb.archive.ubuntu.com/ubuntu/ 45Gb 1.5Gb 256Mb
oolong3.uta.fi 153.1.17.87 etch http://ftp.fi.debian.org/debian/ 30Gb 512Mb 256Mb
oolong4.uta.fi 153.1.17.88 dapper http://gb.archive.ubuntu.com/ubuntu/ 20Gb 256Mb 256Mb

Chapter 5: oolong2.uta.fi

oolong2.uta.fi hosts the Gamelab's website, the administrator is Jussi Kuittinen.

Chapter 6: oolong3.uta.fi

oolong3.uta.fi hosts the web services related to Gamelab's researches, including a movabletype blog system (http://www.pelitutkimus.fi) and a mediawiki wiki system (http://www.gamescultures.org) so far. This chapter describes the web services running on oolong3.uta.fi, their related configuration, maintance and backup policy.

The web services on oolong3.uta.fi are built upon: Apache web server, MySQL database system, Movabletype, and Mediawiki. The versions are listed as below:

Fig6. Software Information
Software version
Apache 2.2.3
MySQL 5.0
Movabletype MTOS 4.1
Mediawiki 1.11.1

6.1 Database Management

Database management can be done with web interface, the URL is http://oolong3.uta.fi/phpmyadmin/. The database name and user name are listed below:

Fig7. Database Credentials
user database password
root root can access and manipulate all databases
pelitutkimus mt to be filled
gamescultures mediawiki to be filled

6.2 Apache Configuration

We use Apache virtual host feature to host different website with different domain names, each website has its own configuration. The configuration files are place at /etc/apache2/sites-available, pelitutkimus is for pelitutkimus.fi and gamescultures is for gamescultures.org.

Debian has its special directory design for virtual host managment. To add a new website:

6.3 Backup Policy

The backup directory on oolong3 is /backup. There are two backup scripts doing daily backup: /etc/cron.daily/automysqlbackup and ~/root/back/back.pl. /etc/cron.daily/automysqlbackup creates daily, weekly and monthly database backup and stores them in /backup/db, ~/root/backup/backup.pl backups /backup/db, /etc/apache2, /var/www/pelitutkimus.fi, and /var/www/gamescultures.org, and copies the generated tar.gz file to the network hard disk drive //arnika/laitos$/hyper/Projektit/oolong3_Backup/ as remote backup.

On oolong3, the network hard disk drive arnika is auto mounted at booting to /mnt/arnika, where /mnt/arnika/hyper/Projektit/oolong3_Backup/ is a hidden directory that is only visible to Frans, Mikael and Chienting, any permission changes has to be configured by Mikael. To change the auto mount configuration, edit the file /etc/fstab and /etc/arnikapw. By now, the username and password used for connecting //arnika is Chienting's username and password, they will have to be changed in the future.

6.4 Pelitutkimus.fi

pelitutkimus.fi is a website providing information about game, game research and game design in Finnish. It is built by Movable Type Open Source 4.1 (MTOS4.1), with Mollio as its design template.

The configuration file of Movable Type is /var/www/pelitutkimus.fi/mt/mt-config.cgi, it is automatically generated when installing Movable Type, it can also be manually modified to further meet our requirement, just remember to backup before making any changes to the file. Be sure to keep this file safe, because it also contains the database username and password used by Movable Type. Information on all configuration options can be found at http://www.movabletype.org/documentation/config. To manage pelitutkimus.fi, connect to log in page, the administrator username and password are: admin / (to be filled).

The Movable Type User and Administration Manual is stored at //arnika/hyper/Projektit/Gamelab/Server_documents/MovableType, more documentation can be found at movabletype.org/documentation.

6.5 Gamescultures.org

gamescultures.fi is a website for creating knowledge about games cultures. It is built by Mediawiki 1.11.1. The configuration file of Mediawiki is /var/www/gamescultures.org/wiki/LocalSettings.php, it is automatically generated when installing Mediawiki, it can also be manually modified to further meet our requirement, just remember to backup before making any changes to the file. Be sure to keep this file safe, because it also contains the database username and password used by Mediawiki. Information on all configuration options can be found at http://www.mediawiki.org/wiki/Manual:Configuration_settings. To manage gamescultures.org, connecto to log in page, the administrator username and password are: admin / (to be filled).

In addition to the basic Mediawiki, there are extra plugins installed and mofications made to improve the functions.

Short URL turns the original long URL http://www.gamescultures.org/index.php/Main_Page to a shorter one, http://www.gamescultures.org/Main_Page. To achieve this, follow the instructions on mediawiki website. For oolong3, there are two files to modify, /etc/apache2/sites-available/gamescultures and /var/www/gamescultures.org/wiki/LocalSettings.php. In /etc/apache2/sites-available/gamescultures, added the following content:

RewriteEngine On
# RewriteBase /wiki

RewriteCond %{REQUEST_URI} !^/(stylesheets|images|skins)/
RewriteCond %{REQUEST_URI} !^/(redirect|texvc|index).php
RewriteCond %{REQUEST_URI} !^/error/(40(1|3|4)|500).html
RewriteCond %{REQUEST_URI} !^/favicon.ico
RewriteCond %{REQUEST_URI} !^/robots.txt

# Rewrite http://wiki.domain.tld/article properly, this is the main rule
RewriteRule ^/(.*)$ /index.php/?title=$1 [L,QSA]    
    

Then added the following content to /var/www/gamescultures.org/wiki/LocalSettings.php:

## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
## For short URL
## short urls
$wgArticlePath      = "/$1";
$wgUsePathInfo = false;
    

File upload allows users to upload files to gamescultures.org. To enable this feature, add or modify the following options in /var/www/gamescultures.org/wiki/LocalSettings.php:

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads       = false;
    

If you want to have a preview of the uploaded image, you have to first install ImageMagick, then add or modify the folloing options in /var/www/gamescultures.org/wiki/LocalSettings.php:

$wgUseImageMagick = true;
$wgImageMagickConvertCommand = "/usr/bin/convert";    
    

Extra Edit buttons help users edit content on gamescultures.org, so they don't have to memorise too many complex wikipedia syntax. There are more than one way to do it, the way gamescultures.org does it to add the javascript of extra edit buttons to http://www.gamescultures.org/MediaWiki:Common.js. In order to do so, you must login as an administrator.

6.6 Domain Name management

The registrar of both pelitutkimus.fi and Gamescultures.org is Nebula, any domain name related changes (ex. @ record, CNAME record, MX record) is done via Nebula customer service emai: yritystuki@nebula.fi.

Both domain names use Google applications for you domain to provide email service. To manage Google applications for you domain service, login from Google applications for you domain page, usernames and passwords are as below:

Fig8. Account information of Google applications for you domain
Domain Username Password
pelitutkimus.fi admin@pelitutkimus.fi Frans knows it
gamescultures.org admin@gamescultures.org To be filled

Chapter 7: oolong4.uta.fi

oolong4.uta.fi hosts the web services related to the Social Media research group, it is a mediawiki wiki site at this moment. The system administrator is Jani.