Data Grid HDF5 Virtual File Driver

The Data Grid Virtual File Driver enables NCSA's HDF5 library to process HDF5 files living on the Data Grid.
Access to remote files is provided by the Data Grid Storage Client API package from Argonne National Labs.
This API interfaces to different Data Grid Storage Servers which can be a Distributed Parallel Storage System (DPSS)
or any Web browser (for the moment only DPSS supports partial file access, a WWW Storage Client driver will be added soon).

How to use it

As long as it isn't yet integrated into the HDF5 distribution, the Data Grid Virtual File Driver is available as a patch  against
the orginal HDF5 snapshot version 1.3.5 which can be found  here . The patch also includes some simple demo programs
in the test/ subdir: datagrid_read.c anddatagrid_write.c. Have a look at these to learn how to activate the Data Grid driver
in your application before opening a Data Grid file. That's the only difference - every following HDF5 call behaves as you
would operate on a local HDF5 file.

How to get and install it

Because it builds on top of quite a lot of other packages, here is a list of software components you need for installing an
HDF5 library with the Data Grid extensions:
 

• OpenLDAP-1.2.7-globus, SSLeay-0.9.0b, globus-1.1.1

 

 

Just follow the installation instructions from the Globus pages
 
 

• Berkeley db-2.7.7

 

 

This library is already included in the DPSS sources.

cd build_unix
../dist/configure
make install
 
 

• gdbm-1.8.0

 

 

RedHat 6.x already has this library. Otherwise this comes also with the DPSS sources.

./configure
make
make install
 

• DPSS-2.6

 

 

Get the  binaries  directly, or get the  sources  and build with

cd build_<arch>
../configure  --with-db=<db-path> --with-ldap-dir=<ldap-path> --prefix=<install-path>
make libinstall
make install
 

• grid_storage-0.8

 

 

Newest version from ANL can be found here.
My  installation is based on grid_storage-0.8. Build it with

mkdir BUILD
../configure --with-dpss-path=<dpss-path> --with-db-path=<db-path> --with-drivers=file,dpss
make
make install
 

• hdf5-1.3.5

 

 

Get the original HDF5 1.3.5 snapshot, apply the patch, and build the stuff

tar xzf hdf5-1.3.5.tar.gz
cd hdf5-1.3.5
patch -p1 < ../hdf5-1.3.5-datagrid.patch
autoconf
configure --with-gass=<globus-development-path> --with-ssl-libpath=<ssl-path>/lib --with-datagrid=<datagrid-path> --with-db-libpath=<db-path>/lib --with-dpss-libpath=<dpss-path>/lib
make
make install

Check out the Data Grid examples programs in the test/ directory.

./datagrid_write file://localhost/tmp/bla.h5
./datagrid_read file://localhost/tmp/bla.h5

The first program writes a simple HDF5 file using the Data Grid VFD. The second one
just reads this file back in. If you have access to a DPSS, replace the URL by
x-dpss://dpss-host/bla.h5.