10 Mbps Advanced Packet Vault

Operations Guide

Center for Information Technology Integration
August 2001

Introduction

Project Overview

Hardware Environment

The SCSI disk is connected to the Ultra 160 channel of the on-board SCSI controller, and the tape library changer and LTO tape drive are connected to the Adaptec 29160 SCSI controller. The SCSI disk is used exclusively for buffering packets before they are written to tape, storing packets retrieved from tape, and for the APV database; the IDE disk is used for all other permanenent storage. Not shown in the above table are standard components including a monitor, keyboard, mouse, and floppy and CD-ROM drives.

OpenBSD 2.9 currently does not utilize the second processor, but this is promised.

Software Environment

Installing OpenBSD

We recommend the following IDE disk partition sizes. The SCSI disk partition will be created later.
 

/	1 GB
/tmp	1
/var	2
/usr	4
/home	5
swap	1

When choosing which installation packages to install, we recommend you select all packages; when asked if you ever plan to run X, we recommend you respond affirmatively.

After verifying correct system operation, you may proceed by making APV kernel and system program modifications; obtaining, building, and installing prerequisite packages; and compiling the APV user programs.
 

Unpacking the APV Source Distribution

cd /usr
tar zxf apv10.tar.gz

In the remainder of this document, file and directory names beginning with apv are assumed to refer to the root of the APV source distribution (/usr/apv in this example).

The top level of the APV software distribution is organized as follows:
 

bin	APV OpenBSD system program modifications.
bpf	Utility routines for reading BPF-format data.
crypto	Utility routines for encrypting and decrypting data.
crypto/rijndael	Utility routines for AES.
decrypt	User program to decrypt data from the vault.
des	Utility routines for DES.
doc	APV documentation and OpenBSD configuration files.
dump	User program to encrypt data.
listen	User program to read data from network.
pilot	Scripts for running the APV.
sys	APV OpenBSD kernel modifications
util	Utility routines.

Building a modified OpenBSD Kernel

Installing APV Source Distribution Kernel Files

The APV source distribution contains a patch file apv/sys/kernel_patch that will apply the above changes.  Apply the patch by entering the following commands with root authority:

cd /usr/src/sys
patch <apv/sys/kernel_patch

Kernel Compilation Options

We had to disable the IPv6 options for our STL2 server board's on-board Ethernet controller to work. If you have a different motherboard, you may not need to disable these options, otherwise disable the following:
 

INET6	Disable IPv6 kernel support.
PULLDOWN_TEST	Requires IPv6.

The following pseudo-devices must be disabled:
 

gif	Requires IPv6.

The APV source distribution includes appropriately modified configuration files apv/sys/arch/i386/conf/VAULT4 and apv/sys/conf/GENERIC4, which should be copied to the corresponding places in the kernel build directory /usr/src/sys.

Building the Kernel

cd /usr/src/sys/arch/i386/conf
config ./VAULT4
cd ../compile/VAULT4
make depend && make && cp bsd /

Later you will be asked to reboot your system to activate the new kernel.

Updating /usr/include

cd /usr/src/sys/net
cp bpf.h bpfdesc.h /usr/include/net
 

Building modified OpenBSD System Utilities

The APV source distribution contains a patch file apv/bin/user_patch that will apply the above changes.  Apply the patch and build and install a modified chio by entering the following commands with root authority:

cd /usr/src/bin/
patch <apv/bin/user_patch
cd chio
make
cp obj/chio /bin
 

Obtaining and Building Prerequisites

Berkeley DB

gnupg

pkg_add gnupg-1.0.6.tgz
 

tcl

pkg_add tcl-8.3.2.tgz

If you have the ports collection installed, you can install tcl just by issuing the following commands as root:

cd /usr/ports/lang/tcl/; make install

and making a symbolic link:

ln -s /usr/local/bin/tclsh8.3 /usr/local/bin/tclsh
 

gnutar

cd /usr/local/src
tar xzf tar-1.13.19.tar.gz
cd tar-1.13.19
./configure
make

After successfully building the program, run the following with root authority, which will install the gnu tar program as /usr/local/bin/tar.

make install

Compiling the APV User Programs

cd apv
make
 

Preparing to run the APV

Set Up Directories and Filesystems

NOTE:  For host platforms whose components differ from ours (see the "Hardware Environment" section above) the name of the SCSI disk device may be differ from /dev/sd0a!

newfs /dev/sd0a
ed /etc/fstab
$a
/dev/sd0a /scratch0 ffs rw 1 2
.
w
q

Create directories in and links to the partition:

mkdir /scratch0
mount /scratch0
cd /scratch0
mkdir volumes retrieved fileDB

Create a directory /mfs, which the vault will use as a mount point for the mfs filesystem:

mkdir /mfs
 

Remove Daemons and Periodic Tasks

sendmail_flags=NO
inetd=NO
lpd=NO

Later you will be asked to reboot your system to activate the above changes.

OpenBSD runs several tasks periodically, and they will consume resources needed by the APV. Issue the following command as root:

crontab -e

and delete the lines specifying daily, weekly, and monthly maintenance.

NOTE: it is important that these scripts be run periodically, especially /etc/daily. We recommend that these scripts be run manually, at a time when the APV is not running.

Ethernet Device Name

To determine the name of the network interface, use the command

netstat -in

to list the available network interfaces. Once this has been determined, change the apv/pilot/pilot.tcl script to use the new name:

ed apv/pilot/pilot.tcl
/set interface/
s/fxp0/newname/
w
q
 

Panic Action

ed /etc/sysctl.conf
/ddb.panic/
s/^/#/
w
q
 

Reboot System

reboot

Inspect the boot log by typing

dmesg

and examining the output for signs that the SCSI subsystem has recognized the tape library changer (ch0 device) and the library tape drive (st0 device).

If all is well, log in under the new kernel and issue the following command with root authority:

chio status

You should see output of the form

picker 0: 
slot 0: <ACCESS,FULL> [PVT=A0000002                        ]
slot 1: <ACCESS,FULL> [PVT=A0000004                        ]
slot 2: <ACCESS>
slot 3: <ACCESS>
slot 4: <ACCESS,EXCEPT,FULL> [PVT=] [ASC=0x83,ASCQ=0x09]
slot 5: <ACCESS>
slot 6: <ACCESS,FULL> [PVT=A0000000                        ]
slot 7: <ACCESS>
slot 8: <ACCESS>
slot 9: <ACCESS>
slot 10: <ACCESS>
portal 0: <INEAB,EXENAB,ACCESS>
drive 0: <FULL> [PVT=A0000003                        ]

Load Tapes

Create Master Key

The master key is identified by the User-ID you select for it (see below). It is important that you select a name that is distinct from that of all other vaults. For this reason, specify an email address using the fully-qualified domain name of the host platform.

Become root, and ensure that any previous keypairs are deleted:

/bin/rm -rf /root/.gnupg

Create the master key:

gpg
gpg --gen-key

The first invocation initializes things, the second creates tke keypair. When asked, select the default kind of key (DSA and ElGamal); the default key size (1024 bits); a key lifetime of 0 (does not expire); and for the key User-ID specify real name "apv10", email address "apv10@apv.your.domain.name, and comment "kv1". Next, choose a passphrase to protect the key. The passphrase will be required for retrieving vault data.

After the keys are generated, you can display some information about them using:

gpg --list-public-keys
gpg --list-secret-keys
 

Starting the APV

cd apv/pilot
./apvstart -l

Start the archiver by issuing the following commands with root authority in another window:

cd apv/pilot
./apvstart -a
 

Stopping the APV

cd apv/pilot
./apvstart -l

The archiver will suspend when there are no more volumes to write to the tape, and will resume when more volumes appear.

The archiver may be stopped by typing Ctrl-C in its window. The recommended way of stopping the archiver is to stop the listener first, waiting until all data have been written to tape, and then stopping the archiver. If the archiver is currently writing a volume to tape when it is requested to stop, it will not stop immediately but will finish writing the volume to the tape first. This process can take several minutes; the archiver will display progress messages in its window. At exit, the archiver unloads the tape from the drive and stores it in the library. Any volumes waiting to be written to tape will be written to a new tape when the archiver is next started.

To assist in determining whether a volume is being written or if any volumes are waiting to be written, execute the following commands with root authority:

cd apv/util
./apvsync status

A volume with status "FILLED" is waiting to be written to tape, but the write operation has not begun; status "DRAINING" indicates the volume is being written to tape.

Monitoring the APV

• listen is responsible for sniffing packets from the network and writing them out to files (called segments) in MFS.
• pkt_dump reads the segments written by listen, encrypts them, and writes the results to disk.
• pilot.tcl oversees the operation of listen and pkt_dump, and gathers the segments written by pkt_dump into volume directories (in /scratch0/volumes/).
• archiver.pl writes complete volumes out to tape.

To make sure the logs are regularly rotated, lines should be added to the file /etc/newsyslog.conf; an example is provided in apv/doc/newsyslog.conf. See man newsyslog for more information.

It may also be interesting to watch the output of df /mfs and df /scratch0. In normal operation, the space used in each filesystem should be relatively stable. However if, for example, network traffic significantly exceeds 10 Mbps, /mfs will fill up as pkt_dump lags behind, and /scratch0 will fill up as archiver.pl lags behind.

Database

Recovering Data from the APV

Data may be recovered from a tape by issuing the following commands with root authority:

cd apv/pilot
./retrieve.pl [--tapeid=id --firstvol=first --lastvol=last]

where tapeid is the ID from the label of the desired tape, firstvol is the sequence number of the first volume to be recovered (there will be at most 100 volumes per tape; the first volume has sequence number one), and lastvol is the number of the last volume. If the last two parameters are elided, all volumes on the tape are restored.

Alternatively, if the retrieve.pl command is given without the tapeid parameter, you will be prompted to enter the tape id and, optionally, the starting and ending volume numbers. When the specified tape has been processed, you will be prompted for another.

retrieve.pl will demand the passphrase used to encrypt the vault master key; this must be entered twice to prevent accidental mistyping. Terminal echo is turned off.

The specified tape will be loaded, and the specified volume(s) will be copied from tape, decrypted, and stored in directory /scratch0/retrieved. Each volume directory will consist of a number of decrypted segment packet files (named :n.d, where n denotes the epoch at which the last packet was returned by BPF for the segment, and d disambiguates files written at the same epoch) and segment descriptor files (named x:n.d). Each segment packet file is in tcpdump format and may be viewed with the command

tcpdump -r :n.d

The associated descriptor file is a text file giving more details about the data in the segment:
 

starttime	Epoch at which the first packet was returned by BPF.
endtime	Epoch at which the last packet was returned by BPF.
lis_pkts	Number of packets returned by BPF.
lis_pkts_drop	Number of packets dropped by BPF.
lis_bytes	Number of bytes read by BPF.
lis_bytes_snap	Number of bytes returned by BPF.
lis_bytes_drop	Number of bytes dropped by BPF.
lis_bytes_per_sec	Bytes/sec processed by the listener.
dmp_pkts	Number of packets read by the dumper.
dmp_pkts_written	Number of packets written by the dumper.
dmp_bytes	Number of bytes read by the dumper.
dmp_bytes_written	Number of bytes written by the dumper.
dmp_bytes_per_sec	Bytes/sec processed by the dumper.

It will not be possible to use the APV for writing volumes to tape while it is being used to recover data, although the APV listener subsystem can be running concurrently. However, resource contention between the listener and the retrieval process could force the listener to stop.

Understanding and Recovering from APV Errors

Packet Overrun

If such overruns are noticed, you may want to review the use of system resources (see "Remove Daemons and Periodic Tasks") to verify that periodic tasks or daemons are not the cause of lost packets.

MFS/UFS Exhaustion

In a similar vein, the listener subsystem will stop if less than 1 GB are found to be available in UFS. Again, this is noted in the log and the APV will stop collecting packets from the network. This may be caused by the archiver waiting for scratch tapes, or continuous heavy traffic with which the archiver cannot keep up.

If either of these conditions are encountered, more monitoring of the system resources may indicate where the problem lies. In either case, the listener subsystem may be restarted after the cause has been determined and the backlog has been alleviated.

Tape Exhaustion

Unexpected Failures

Host Platform Crashes

cd apv/pilot
./reconcile.pl

These commands will reconcile the state of the buffered volumes in UFS with the archiver.  When the archiver is next restarted, these buffered volumes will be written to tape.

Advanced configuration

Examination of the apvstart script will show that the main job of apvstart -l is to run the shell script pilot.sh, which in turn runs pilot.tcl. Both pilot.sh and pilot.tcl accept a number of command-line options; run pilot.tcl --help to get a list. You can use any of these options by appending them to the line of apvstart in which pilot.sh is run.

In particular, command-line options can be used to choose any of three different file formats for the encrypted files that the vault produces: conversation format (the default), endpoint format, and the format used by the prototype vault; see [ACF01] for a description of the differences between these formats.

Resource requirements

Memory space

Disk space

UFS disk space

Conversation and endpoint modes add different amounts of extra data to each packet. In the worst case, pkt_dump in conversation mode might produce 102 MB in UFS per minute, and in endpoint mode might produce 135 MB per minute. However, the data won't be written to tape until an entire 1 GB volume is completed. The above memory and disk space recommendations will ensure there is enough space to buffer several volumes.

Database disk space

Log disk space

Tape consumption

To assist with tape management, use the following commands.

Tape Status

cd apv/pilot
./tapestatus.pl

In the output of this command, "WRITTEN" means the tape has been written to and can be removed; "SCRATCH" denotes a fresh tape available for writing; "N/A" means an empty or unavailable slot.

Removing Filled Tapes

chio move slot n portal 0

The tapestatus.pl output will identify the slot numbers, n, of tapes that have been written. Use chio to move each tape in turn to the I/O port; press and hold the "*" key on the library front panel and then press "MENU" to open the I/O port and eject the tape. Press the "*" and "MENU" key sequence again to close the port.

Ingesting Fresh Tapes

chio move portal 0 slot n

where n is the number of an empty slot identified as "N/A" by the output of tapestatus.pl. For each fresh tape in turn press and hold the "*" key on the library front panel and then press "MENU" to open the I/O port, and insert the tape into the port, after which the port will close automatically. Use chio to move the tape from the I/O port to an available slot.

Bulk Moves

References

[Qua01] Qualstar Corporation, "TLS-6000 SCSI-2 Interface Manual," 501205 Revision A. http://www.qualstar.com, under "Technical Services."