Data Back-up Requirement

Last year (2008), when getting processes in place for our new web design and software development business, getpepper™, I put together an all important data back-up procedure. My aim was, in the worse case, to ensure that we could restore all but the last few days-worth of data for our own systems and that of our clients.

We mostly use open source systems and tools to create and manage our websites and write our software, with the Ubuntu Linux distribution forming the platform upon which our server-based systems (source control (Subversion), issue tracking (Trac), CRM, etc.) and some of our desktop systems run.

After some research on Linux-based back-up facilities I settled on using a (Bash) shell script that would allow us to run a regular back-up cycle using external portable hard disks. The weekly back-up shell script, which saves to USB hard drives, is based upon a script provided by Mike Rubel.

The Script

The script uses the rsync tool to provide the incremental functionality of the back-up. Here's the adapted script that we use.

# ============================
# Author: Paul Pepper (though see description below for credits)
# Created: 6 November 2008
# Description:
#   Rotating-snapshot utility adapted from Mike Rubel's make_snapshop.sh which
#   can be found at http://www.mikerubel.org/computers/rsync_snapshots/
#   Basically, this script performs rotating backup-snapshots of /home whenever
#   it is called.
# ============================


#!/bin/bash

unset PATH # suggestion from H. Milz: avoid accidental use of $PATH

# ============================
# System commands used by this script
# ============================

ID=/usr/bin/id
ECHO=/bin/echo

MOUNT=/bin/mount
UMOUNT=/bin/umount
RM=/bin/rm
MV=/bin/mv
CP=/bin/cp
TOUCH=/bin/touch
RSYNC=/usr/bin/rsync

# ============================
# File names and locations
# ============================

MOUNT_DEVICE=/dev/sdb1
MOUNT_POINT=/media/sdb
BACKUP_TO_DIR=/backup/
BACKUP_FROM_DIR=/home
SNAPSHOT=snapshot
EXCLUDES=
BACKUP_TO_PATH=${MOUNT_POINT}/${BACKUP_TO_DIR}/${SNAPSHOT}

# ============================
# The script
# ============================

# Make sure we're running as root
if [ `$ID -u` != '0' ]; then
    $ECHO "$0 must be executed as root.  Exiting!"
    exit 1
fi

# Attempt to mount the backup device, else abort
$MOUNT -o rw $MOUNT_DEVICE $MOUNT_POINT
if [ $? -ne 0 ]; then
    $ECHO "$0: Could not mount $MOUNT_DEVICE on $MOUNT_POINT as readwrite"
    exit 1
fi

# Step 1: delete the oldest snapshot, if it exists:
if [ -d ${BACKUP_TO_PATH}.3 ] ; then
    $RM -rf ${BACKUP_TO_PATH}.3
fi

# Step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d ${BACKUP_TO_PATH}.2 ] ; then
    $MV ${BACKUP_TO_PATH}.2 \
        ${BACKUP_TO_PATH}.3
fi

if [ -d ${BACKUP_TO_PATH}.1 ] ; then
    $MV ${BACKUP_TO_PATH}.1 \
        ${BACKUP_TO_PATH}.2
fi

# Step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d ${BACKUP_TO_PATH}.0 ] ; then
    $CP -al ${BACKUP_TO_PATH}.0 \
            ${BACKUP_TO_PATH}.1
fi

# Step 4: rsync from the system into the latest snapshot (notice that
# rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first.  If it were not so, this would copy over the other
# snapshot(s) too!
$RSYNC -va --delete --delete-excluded     \
    --exclude-from="$EXCLUDES"            \
    ${BACKUP_FROM_DIR} ${BACKUP_TO_PATH}.0

# Step 5: update the mtime of our most recent snapshot.
$TOUCH ${BACKUP_TO_PATH}.0

# Unmount the device to which we've written the backup
${UMOUNT} ${MOUNT_POINT}
if [ $? -ne 0 ]; then
    $ECHO "$0: Could not unmount ${MOUNT_POINT}"
    exit 1
fi

Here's an outline of our weekly back-up process:

1. Grab one of the external hard disks.
2. Attach hard disk to server via a USB connector.
3. Log in to server as a regular user - don't su to root!
4. Run the back-up shell script:

   $ sudo ./backup-snapshot.sh
           

5. Disconnect hard disk and return to its place of safe-keeping!

And that's it!

All users who perform the back-up must have the necessary permissions to run the shell. I enforce this by adding those users as sudoers, only permitting privileged access to the back-up shell script. Here's the relevant parts of the sudoers file that grants that access - note that it is recommended that you use visudo when editing this file.

# User alias specification
User_Alias  BACKUP_USERS = ann, bill

# Cmnd alias specification
Cmnd_Alias  BACKUP_CMND = /somepath/backup-snapshot.sh, /bin/mount, /bin/umount, /bin/touch, /usr/bin/rsync

# Permit BACKUP_USERS to run the back-up script as root from all locations
BACKUP_USERS    ALL=(root) BACKUP_CMND

Labels: back-up, linux, shell-script

In order to access a persistence store using JDBC, it's necessary to load the JDBC driver using the Class class's static method forName(). Once the JDBC driver has been loaded, it should be possible to make a connection to a database managed by the DBMS. Here's how the postgresql JDBC driver is loaded (don't forget to make sure the JDBC driver jar file is either on the class path, or referenced directly), and then used to get a database connection:

// Load the driver class
Class.forName("org.postgresql.Driver");

// Obtain a connection to the DBMS
java.sql.Connection connection =
    java.sql.DriverManager.getConnection(
        "jdbc:postgresql://localhost/dbname",
        "username",
        "password");

Once you have a connection you can start manipulating the database. Use the java.sql.Statement class to do this with the java.sql.ResultSet to manage the results of executing a statement.

There are two other types of statement available, namely CallableStatement and PreparedStatement, which offer stored procedure statements and preparsed SQL statements, repectively. These have the advantage of reducing the overhead of parsing executed SQL at run-time. The PreparedStatement is likely the more useful due to its simplicity and improved efficiency. Here's an example of its use:

try {
    PreparedStatement ps = null;

    ps = c.prepareStatement("INSERT INTO authors VALUES (?, ?, ?)");
    ps.setInt(1, 495);
    ps.setString(2, "Light-Williams");
    ps.setString(3, "Corwin");

    ps.executeUpdate();
} catch (SQLException se) {
    System.out.println(
        "We got an exception while preparing a statement:" +
        "Probably bad SQL.");
    se.printStackTrace();
    System.exit(1);
}

After executing an SQL statement, obtain the results set data, including the names and types of columns that have been updated by a statement, using an instance of ResultsSetMetaData obtained from the ResultsSet instance.

Similarly the DatabaseMetaData can be used to obtain information such as the catalogues available from the connected database, the producer of the database and the user who is used for the connection. Get an instance of the DatabaseMetaData via the Connection object instance through the getMetaData() method.

It's worth noting a few points about the results set returned by executing an SQL statement. Firstly the returned ResultsSet object starts off pointing to the the position prior to the first record. This means that the next() method must be called on the ResultsSet object in order to get the first record. Also, there is no way of finding out the number of records held by a returned ResultsSet instance except by stepping through it and counting the number of records. Finally, in multi-threaded applications ensure that each thread uses its own ResultsSet objects.

Labels: java database

Background

These notes relate to Debian-based systems running Apache 2.2, so you’ll have to make the appropriate changes to paths, and possibly commands, for your operating system or Linux distro.

The example setup that I’ve provided here allows users with an operational Apache user directory (mod_userdir) to set their own user access permissions, rather than a system-wide approach.

System-wide Settings

The Apache2 configuration files can be found in /etc/apache2/. Update the file /etc/apache2/apache2.conf to include the following directives:

<Directory /home/*/public_html>
    AllowOverride FileInfo AuthConfig Limit
    Options Indexes SymLinksIfOwnerMatch IncludesNoExec
</Directory>

You might simply be able to uncomment existing text within the config file. Among other things, this permits users to enable authentication checking in their public_html directories, or whatever you set the directory name to. You'll also have to enable Apache's mod_userdir if it isn't already enabled:

$ sudo a2enmod userdir

Support for digest authentication is also provided in an Apache module. The digest authentication module is not enabled by default, but can also be enabled using a2enmod:

$ sudo a2enmod auth_digest

If a2enmod isn't available on your distribution, then you may wish to enable Apache modules by providing a sym link to the appropriate module in the following manner:

ln -s /etc/apache2/mods-available/auth_digest.load /etc/apache2/mods-enable

Password Generation

Passwords are generated using the htdigest tool that ships with the Apache2 distribution. The file created using this tool places username, realm and hashed password together on a colon-delimited line. This file should be placed in a location where Apache cannot serve it up to a client (e.g. don’t place it in /var/www).

In order to add an entry to the password file run the htdigest tool as follows:

$ htdigest -c /directory/path/digest.htpasswd myrealm username

Caution: the -c flag forces htdigest to delete the existing digest password file, if it already exists. Drop its use of you need to add new entries into the file. You should also replace the values myrealm and username with values appropriate for your system. The realm value is a security context that should be recognisable to the user in order to allow them to provide the correct username and password.

Directory-Level Configuration

You can now create a .htaccess file within each of those directories and subdirectories that you would like to be maintain access control to using digest authentication. Here’s an example .htaccess file that may, for example, be placed immediately within a user's public_html directory:

AuthType            Digest
AuthName            "myrealm"
AuthDigestDomain    / http://subdomain.mydomain.com/
AuthUserFile        /directory/path/digest.htpasswd
Require             valid-user

Here's an explanation of each of the above Apache directives.

• The AuthName value is the same value that was given when using the htdigest program (see details above).
• AuthDigestDomain provides the list of URIs that are in the protection space. These URIs can be absolute or relative and sub-directories of those given are matched also.
• The value of AuthDigestFile points to the location of the file that was created using the htdigest tool.
• Require takes two values here, but can take more so that extra requirements are imposed. The values used are used to indicate that the user-level authentication mechanism is being used (rather than group-level) and that only valid users (created using htdigest, as shown above) are granted access.

Now Try It!

Restart apache

$ sudo /etc/init.d/apache2 restart

If all has gone well, you should be challenged for your credentials when you try to browse your protected directories.

Labels: apache, http, security

The Problem - Cropped Background Upon Scrolling

Here's the problem: create a block-level (full page width) element and set either a repeat-x background image or background colour, only to see that background cropped to the width of the browser viewport when performing a horizontal page scroll.

This problem occurs when using a fixed-width block-level element (such as content containers) with other block-level elements (such as headers, menu-bars or footers) that run through the full width of the page. If the page is then viewed with a browser window sized less than the width of the container, then scrolling the container into view displays a cropped header, menu and/or footer.

The following screen-shot shows the problem when viewed in Firefox.

And here's the code used to create the above result.

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <style title="text/css">
* {
    margin: 0;
    padding: 0;
    border: 0;
}
#menu {
    height: 50px;
    background: #0a0;
}
#container {
    height: 50px;
    width: 500px;
    background: #a00;
}
    </style>
</head>
<body>
    <div id="menu">
This is the menu.
    </div>

    <div id="container">
This is the container.
    </div>
</body>
</html>

The Solution - min-width

I've fallen foul of this problem before and seen it occur on plenty of other sites. One way to prevent the cropping problem above is to set a min-width CSS attribute on the element containing the full width background. Amending the #menu id selector in the above code solves the problem:

#menu {
    min-width: 500px; // set to the width of the #container
    height: 50px;
    background: #0a0;
}

I've found that applying the min-width attribute to the body also ensures the backgrounds of all non-width-constrained block-level elements extend to the full width of the page, rather than just the non-scrolled viewport.

This seems to apply to all the browsers that I've checked so far, so it's likely W3C stardard-compliant behaviour, but it does seem somewhat unintuitive.

Labels: css, html