|Intel STL2 Server Board||Dual 866 Mhz Pentium III processors, dual PCI (33 Mhz/32 bit, 66 Mhz/64 bit)|
|Adaptec AIC-7899 U160 (on-board STL2)||Dual-port SCSI controller; provides separate Ultra160 and Ultra Wide SCSI channels|
|Intel EtherExpress Pro100+ (on-board STL2)||10/100 Mbps PCI Ethernet controller|
|512 MB RAM (on-board STL2)||Two Kingston KVR133X72RC3/256-IS PC-133 256 MB memory modules|
|Adaptec 29160 SCSI Controller||single-channel Ultra 160 SCSI controller|
|35 GB SCSI Disk||Seagate ST336704LWV Cheetah 35 GB SCSI3 disk|
|15 GB IDE Disk||Western Digital WD153AA Caviar 15 GB 5400 RPM Fast ATA/Enhanced IDE compatible|
|Qualstar Tape Library||Qualstar TLS-8211 tape library subsystem with one HP Model 230 LTO tape drive, eleven media slots, a media changer, and an I/O port|
The SCSI disk is connected to the Ultra 160 channel of the on-board SCSI controller, and the tape library changer and LTO tape drive are connected to the Adaptec 29160 SCSI controller. The SCSI disk is used exclusively for buffering packets before they are written to tape, storing packets retrieved from tape, and for the APV database; the IDE disk is used for all other permanenent storage. Not shown in the above table are standard components including a monitor, keyboard, mouse, and floppy and CD-ROM drives.
OpenBSD 2.9 currently does not utilize the second processor, but this is promised.
We recommend the following IDE disk partition sizes. The SCSI
disk partition will be created later.
When choosing which installation packages to install, we recommend you select all packages; when asked if you ever plan to run X, we recommend you respond affirmatively.
After verifying correct system operation, you may proceed by making
APV kernel and system program modifications; obtaining, building, and installing
prerequisite packages; and compiling the APV user programs.
tar zxf apv10.tar.gz
In the remainder of this document, file and directory names beginning with apv are assumed to refer to the root of the APV source distribution (/usr/apv in this example).
The top level of the APV software distribution is organized as follows:
|bin||APV OpenBSD system program modifications.|
|bpf||Utility routines for reading BPF-format data.|
|crypto||Utility routines for encrypting and decrypting data.|
|crypto/rijndael||Utility routines for AES.|
|decrypt||User program to decrypt data from the vault.|
|des||Utility routines for DES.|
|doc||APV documentation and OpenBSD configuration files.|
|dump||User program to encrypt data.|
|listen||User program to read data from network.|
|pilot||Scripts for running the APV.|
|sys||APV OpenBSD kernel modifications|
|Enable direct copy of BPF packets to disk, bypassing user space copies.|
|apv/sys/scsi/ch.c||Enable bar- and ASC/ASCQ codes to be returned from Qualstar TLS-8211 changer in response to CHIOGSTATUS ioctl.|
|apv/sys/scsi/scsiconf.c||Obtain correct response to SCSI inquiry of Qualstar TLS-8211 changer.|
The APV source distribution contains a patch file apv/sys/kernel_patch that will apply the above changes. Apply the patch by entering the following commands with root authority:
|CITI_PACKET_VAULT||Enable CITI BPF, SCSI, and changer modifications.|
|SCSI_2_DEF||Required for tape library.|
|MFS||Memory file system.|
We had to disable the IPv6 options for our STL2 server board's on-board
Ethernet controller to work. If you have a different motherboard, you may
not need to disable these options, otherwise disable the following:
|INET6||Disable IPv6 kernel support.|
The following pseudo-devices must be disabled:
The APV source distribution includes appropriately modified configuration files apv/sys/arch/i386/conf/VAULT4 and apv/sys/conf/GENERIC4, which should be copied to the corresponding places in the kernel build directory /usr/src/sys.
make depend && make && cp bsd /
Later you will be asked to reboot your system to activate the new kernel.
cp bpf.h bpfdesc.h /usr/include/net
|bin/chio||Display tape bar- and ASC/ASCQ codes.|
The APV source distribution contains a patch file apv/bin/user_patch that will apply the above changes. Apply the patch and build and install a modified chio by entering the following commands with root authority:
cp obj/chio /bin
If you have the ports collection installed, you can install tcl just by issuing the following commands as root:
cd /usr/ports/lang/tcl/; make install
and making a symbolic link:
ln -s /usr/local/bin/tclsh8.3 /usr/local/bin/tclsh
tar xzf tar-1.13.19.tar.gz
After successfully building the program, run the following with root authority, which will install the gnu tar program as /usr/local/bin/tar.
NOTE: For host platforms whose components differ from ours (see the "Hardware Environment" section above) the name of the SCSI disk device may be differ from /dev/sd0a!
/dev/sd0a /scratch0 ffs rw 1 2
Create directories in and links to the partition:
mkdir volumes retrieved fileDB
Create a directory /mfs, which the vault will use as a mount point for the mfs filesystem:
Later you will be asked to reboot your system to activate the above changes.
OpenBSD runs several tasks periodically, and they will consume resources needed by the APV. Issue the following command as root:
and delete the lines specifying daily, weekly, and monthly maintenance.
NOTE: it is important that these scripts be run periodically, especially /etc/daily. We recommend that these scripts be run manually, at a time when the APV is not running.
To determine the name of the network interface, use the command
to list the available network interfaces. Once this has been determined, change the apv/pilot/pilot.tcl script to use the new name:
Inspect the boot log by typing
and examining the output for signs that the SCSI subsystem has recognized the tape library changer (ch0 device) and the library tape drive (st0 device).
If all is well, log in under the new kernel and issue the following command with root authority:
You should see output of the form
picker 0: slot 0: <ACCESS,FULL> [PVT=A0000002 ] slot 1: <ACCESS,FULL> [PVT=A0000004 ] slot 2: <ACCESS> slot 3: <ACCESS> slot 4: <ACCESS,EXCEPT,FULL> [PVT=] [ASC=0x83,ASCQ=0x09] slot 5: <ACCESS> slot 6: <ACCESS,FULL> [PVT=A0000000 ] slot 7: <ACCESS> slot 8: <ACCESS> slot 9: <ACCESS> slot 10: <ACCESS> portal 0: <INEAB,EXENAB,ACCESS> drive 0: <FULL> [PVT=A0000003 ]The strings following "PVT=" are the bar-code labels of the corresponding tapes. The ASC/ASCQ codes for the tape in slot 4 indicate the tape has no bar-code. The ASC/ASCQ codes and their meanings are described in Section 14.3.4 of the TLS-6000 SCSI-2 Interface Manual [Qua01] available from Qualstar.
The master key is identified by the User-ID you select for it (see below). It is important that you select a name that is distinct from that of all other vaults. For this reason, specify an email address using the fully-qualified domain name of the host platform.
Become root, and ensure that any previous keypairs are deleted:
/bin/rm -rf /root/.gnupg
Create the master key:
The first invocation initializes things, the second creates tke keypair. When asked, select the default kind of key (DSA and ElGamal); the default key size (1024 bits); a key lifetime of 0 (does not expire); and for the key User-ID specify real name "apv10", email address "email@example.com, and comment "kv1". Next, choose a passphrase to protect the key. The passphrase will be required for retrieving vault data.
After the keys are generated, you can display some information about them using:
Start the archiver by issuing the following commands with root authority in another window:
The archiver will suspend when there are no more volumes to write to the tape, and will resume when more volumes appear.
The archiver may be stopped by typing Ctrl-C in its window. The recommended way of stopping the archiver is to stop the listener first, waiting until all data have been written to tape, and then stopping the archiver. If the archiver is currently writing a volume to tape when it is requested to stop, it will not stop immediately but will finish writing the volume to the tape first. This process can take several minutes; the archiver will display progress messages in its window. At exit, the archiver unloads the tape from the drive and stores it in the library. Any volumes waiting to be written to tape will be written to a new tape when the archiver is next started.
To assist in determining whether a volume is being written or if any volumes are waiting to be written, execute the following commands with root authority:
A volume with status "FILLED" is waiting to be written to tape, but the write operation has not begun; status "DRAINING" indicates the volume is being written to tape.
|dt||Time over which this segment was collected (secs)|
|pr:t(+s)||Packets read (t = total for this listener, s = total for segment)|
|br:t(+s)||Bytes read (t = total for this listener, s = total for segment)|
|bs:t(+s)||Bytes processed (t = total for this listener, s = total for segment)|
|pd:t(+s)||Packets dropped (t = total for this listener, s = total for segment)|
|bd:t(+s)||Bytes dropped (t = total for this listener, s = total for segment)|
|Bps||Bytes/sec processed for this segment|
|Mbps||Mbits/sec processed for thie segment.|
|mf||MFS space free, MB|
To make sure the logs are regularly rotated, lines should be added to the file /etc/newsyslog.conf; an example is provided in apv/doc/newsyslog.conf. See man newsyslog for more information.
It may also be interesting to watch the output of df /mfs and df /scratch0. In normal operation, the space used in each filesystem should be relatively stable. However if, for example, network traffic significantly exceeds 10 Mbps, /mfs will fill up as pkt_dump lags behind, and /scratch0 will fill up as archiver.pl lags behind.
|volid||The volume ID of the volume.|
|tapeid||The ID (bar-code number) of the tape on which the volume was written.|
|sequence#||Sequence number of the volume on the tape (the first volume of the tape has sequence number one).|
|starttime||Epoch at which the first packet of the first segment in the volume was returned by BPF.|
|endtime||Epoch at which the last packet of the last segment in the volume was returned by BPF.|
|segments||The number of segments stored in the volume.|
|lis_pkts||Number of packets in the volume returned by BPF.|
|lis_pkts_drop||Number of packets in the volume dropped by BPF.|
|lis_bytes||Number of bytes in the volume read by BPF.|
|lis_bytes_snap||Number of bytes in the volume returned by BPF.|
|lis_bytes_drop||Number of bytes dropped from the volume by BPF.|
|lis_bytes_per_sec||Bytes/sec processed by the listener for the volume, averaged over all segments.|
|dmp_pkts||Number of packets read by the dumper for the volume.|
|dmp_pkts_written||Number of packets written by the dumper to the volume.|
|dmp_bytes||Number of bytes read by the dumper for the volume.|
|dmp_bytes_written||Number of bytes written by the dumper to the volume.|
|dmp_bytes_per_sec||Bytes/sec processed by the dumper for the volume, averaged over all segments.|
Data may be recovered from a tape by issuing the following commands with root authority:
./retrieve.pl [--tapeid=id --firstvol=first --lastvol=last]
where tapeid is the ID from the label of the desired tape, firstvol is the sequence number of the first volume to be recovered (there will be at most 100 volumes per tape; the first volume has sequence number one), and lastvol is the number of the last volume. If the last two parameters are elided, all volumes on the tape are restored.
Alternatively, if the retrieve.pl command is given without the tapeid parameter, you will be prompted to enter the tape id and, optionally, the starting and ending volume numbers. When the specified tape has been processed, you will be prompted for another.
retrieve.pl will demand the passphrase used to encrypt the vault master key; this must be entered twice to prevent accidental mistyping. Terminal echo is turned off.
The specified tape will be loaded, and the specified volume(s) will be copied from tape, decrypted, and stored in directory /scratch0/retrieved. Each volume directory will consist of a number of decrypted segment packet files (named :n.d, where n denotes the epoch at which the last packet was returned by BPF for the segment, and d disambiguates files written at the same epoch) and segment descriptor files (named x:n.d). Each segment packet file is in tcpdump format and may be viewed with the command
tcpdump -r :n.d
The associated descriptor file is a text file giving more details about
the data in the segment:
|starttime||Epoch at which the first packet was returned by BPF.|
|endtime||Epoch at which the last packet was returned by BPF.|
|lis_pkts||Number of packets returned by BPF.|
|lis_pkts_drop||Number of packets dropped by BPF.|
|lis_bytes||Number of bytes read by BPF.|
|lis_bytes_snap||Number of bytes returned by BPF.|
|lis_bytes_drop||Number of bytes dropped by BPF.|
|lis_bytes_per_sec||Bytes/sec processed by the listener.|
|dmp_pkts||Number of packets read by the dumper.|
|dmp_pkts_written||Number of packets written by the dumper.|
|dmp_bytes||Number of bytes read by the dumper.|
|dmp_bytes_written||Number of bytes written by the dumper.|
|dmp_bytes_per_sec||Bytes/sec processed by the dumper.|
It will not be possible to use the APV for writing volumes to tape while it is being used to recover data, although the APV listener subsystem can be running concurrently. However, resource contention between the listener and the retrieval process could force the listener to stop.
If such overruns are noticed, you may want to review the use of system resources (see "Remove Daemons and Periodic Tasks") to verify that periodic tasks or daemons are not the cause of lost packets.
In a similar vein, the listener subsystem will stop if less than 1 GB are found to be available in UFS. Again, this is noted in the log and the APV will stop collecting packets from the network. This may be caused by the archiver waiting for scratch tapes, or continuous heavy traffic with which the archiver cannot keep up.
If either of these conditions are encountered, more monitoring of the system resources may indicate where the problem lies. In either case, the listener subsystem may be restarted after the cause has been determined and the backlog has been alleviated.
These commands will reconcile the state of the buffered volumes in UFS with the archiver. When the archiver is next restarted, these buffered volumes will be written to tape.
Examination of the apvstart script will show that the main job of apvstart -l is to run the shell script pilot.sh, which in turn runs pilot.tcl. Both pilot.sh and pilot.tcl accept a number of command-line options; run pilot.tcl --help to get a list. You can use any of these options by appending them to the line of apvstart in which pilot.sh is run.
In particular, command-line options can be used to choose any of three different file formats for the encrypted files that the vault produces: conversation format (the default), endpoint format, and the format used by the prototype vault; see [ACF01] for a description of the differences between these formats.
Conversation and endpoint modes add different amounts of extra data to each packet. In the worst case, pkt_dump in conversation mode might produce 102 MB in UFS per minute, and in endpoint mode might produce 135 MB per minute. However, the data won't be written to tape until an entire 1 GB volume is completed. The above memory and disk space recommendations will ensure there is enough space to buffer several volumes.
To assist with tape management, use the following commands.
In the output of this command, "WRITTEN" means the tape has been written to and can be removed; "SCRATCH" denotes a fresh tape available for writing; "N/A" means an empty or unavailable slot.
chio move slot n portal 0
The tapestatus.pl output will identify the slot numbers, n, of tapes that have been written. Use chio to move each tape in turn to the I/O port; press and hold the "*" key on the library front panel and then press "MENU" to open the I/O port and eject the tape. Press the "*" and "MENU" key sequence again to close the port.
chio move portal 0 slot n
where n is the number of an empty slot identified as "N/A" by the output of tapestatus.pl. For each fresh tape in turn press and hold the "*" key on the library front panel and then press "MENU" to open the I/O port, and insert the tape into the port, after which the port will close automatically. Use chio to move the tape from the I/O port to an available slot.
[Qua01] Qualstar Corporation, "TLS-6000 SCSI-2 Interface Manual," 501205 Revision A. http://www.qualstar.com, under "Technical Services."