pNFS/PVFS2 Documentation

Sample PVFS2 Installation Instructions

Note: If you get confused at any point, verify your setup with the diagram below.

Step 1: Download and Untar

• To build PVFS2 and use its kernel module, you need the kernel source. You can either use your own kernel or get the kernel RPMS file.
• Download and untar PVFS2 Server

Step 2: Compile, Boot, and Verify Kernel

• Compile and install kernel on every server. Officially, you only need the same kernel on each client machine (which will load the pvfs2 module), but it sure is easier if all the machines use the same kernel.

Step 3: Compile and Verify PVFS2 storage system

• Compile and install the PVFS2 server code. Follow the instructions on the PVFS2 website, but some ssample instructions are below.
• Cluster Example:

  Switch to root.
  
  Ensure you can ssh without a password to all cluster machines.
  
  Create pvfs2 configuration directory /etc/pvfs2
  
  Create 3 files:
  1. clients - contains hostnames, one per line, of client machines
  2. servers - contains hostnames, one per line, of data server machines
  3. allnodes - contains hostnames, one per line, of all machines except the pvfs2 build client machine you use below.  This file is used to copy the pvfs2 installation to all machines in the cluster.
  
  // On a client machine:
  ./configure -with-kernel=/path/to/kernel -prefix=/usr/local/bin/pvfs2
  make
  make install
  make kmod
  make KMOD_DIR=/usr/local/bin/pvfs2 kmod_install
  
  // Copy installation to every machine:
  cat allnodes | xargs -i sudo scp -r /usr/local/bin/pvfs2 {}:/usr/local/bin
  
  // Define PVFS2 file system and create config files
  /usr/local/bin/pvfs2/bin/pvfs2-genconfig /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf
  
  // Copy /etc/pvfs2 directory to every data server
  cat servers | xargs -i scp -r /etc/pvfs2 {}:/etc
  
  // Create /etc/pvfs2tab file with contents:
  tcp://"mds_server_hostname":3334/pvfs2-fs /mnt/pvfs2 pvfs2 default 0 0
  
  // Copy /etc/pvfs2tab to all client machines
  cat clients | xargs -i scp -r /etc/pvfs2tab {}:/etc
  
  // Create storage space on every data server
  cat servers | xargs -i ssh {} '/usr/local/bin/pvfs2/sbin/pvfs2-server  /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf-{} -f'
  
  // Start server process on every data server
  cat servers | xargs -i ssh {} '/usr/local/bin/pvfs2/sbin/pvfs2-server  /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf-{}'
  
  // Test pvfs2 filesystem using pvfs2 ping tool (Ensure there is no error)
  /usr/local/bin/pvfs2/bin/pvfs2-ping -m /mnt/pvfs2
  
  // In order to mount the pvfs2 file system, first load the pvfs2 kernel module and then start client process.
  cat clients | xargs -i ssh {} /sbin/insmod /usr/local/bin/pvfs2/pvfs2.ko
  cat clients | xargs -i ssh {} /usr/local/bin/pvfs2/sbin/pvfs2-client -p /usr/local/bin/pvfs2/sbin/pvfs2-client-core
  
  cat clients | xargs -i ssh {} mount -t pvfs2 tcp://"mds_server_hostname":3334/pvfs2-fs /mnt/pvfs2
  
  
  // You have successfully created a pvfs2 file system!!  Although you might want to actually verify this....
  

Exporting PVFS2 storage system with NFSv4

Step 0: Ensure NFSv4 works.

• Ensure NFSv4 is working properly. For example, export and mount a local directory through the loopback device. Detailed instructions are located here under Instructions.

Step 1: Setup NFSv4 server

• Pick a pvfs2 client to be exported via the NFSv4 server. Let's call this client the MDS.
• Create /etc/exports file on MDS.

  /mnt/pvfs2  *(rw,sync,fsid=0,insecure,no_subtree_check)
  

• Run commands on MDS:

  rpc.mountd
  rpc.nfsd 8
  exportfs -r
  

Step 2: Mount NFSv4 file system.

• On NFSv4 client:

  mount -t nfs4 "mds_server_name":/ /mnt/nfs4/
  

Debugging Help

• NFS debugging:

  echo 32767 > /proc/sys/sunrpc/nfsd_debug
  echo 32767 > /proc/sys/sunrpc/nfs_debug
  

• PVFS2 debugging, see PVFS2 website

Please help improve these instructions, email dhildebz@umich.edu