dCacche Version: V1.6.6.5
Sample testbed
Testbed machines: dctest01.usatlas.bnl.gov (admin node, dcap door)
dctest02.usatlas.bnl.gov (pnfs node)
dctest03.usatlas.bnl.gov (SRM node)
dctest04.usatlas.bnl.gov (grid-ftp door)
dctest05.usatlas.bnl.gov (read, write pool node)
I: Login and Set up some environment variables
1. Login to all the servers as root
2. In this example, $DCACHE_PKG_PATH will be the temporary directory to place all installation package (Java, Postgres and dCache). You may choose a directory on your server and set up $DCACHE_PKG_PATH.
-----------------------------------------------------------
[root@dctest01 root]# export DCACHE_PKG_PATH=/home/tmp_pkg
[root@dctest01 root]# mkdir $DCACHE_PKG_PATH
[root@dctest01 root]# cd $DCACHE_PKG_PATH
-------------------------------------------------------------
II: Install Java on each node.
The dCache software requires either the JAVA developer kit (jdk) or the runtime environment (jre) to be installed. jdk1.5.0_07 will be installed here.
1. Get the package and install jdk1.5.0_07
---------------------------------------------------------------------------------
[root@dctest01 tmp_pkg]# mkdir $DCACHE_PKG_PATH/jdk-1_5_0_07
[root@dctest01 tmp_pkg]# cd $DCACHE_PKG_PATH/jdk-1_5_0_07
[root@dctest01 jre-1_5_0_04]# wget http://www.atlasgrid.bnl.gov/dcache_admin/pkg/jdk-1_5_0_07/jdk-1_5_0_07-linux-i586-rpm.bin
[root@dctest01 jre-1_5_0_04]# chmod a+x jdk-1_5_0_07-linux-i586-rpm.bin
[root@dctest01 jre-1_5_0_04]# ./jdk-1_5_0_07-linux-i586-rpm.bin
---------------------------------------------------------------------------------
2. Make a symbolic link
------------------------------------------------------------
[root@dctest01 java]# ls -l
total 12
drwxr-xr-x 9 root root 4096 Jan 30 13:46 jdk1.5.0_07
lrwxrwxrwx 1 root root 11 Jun 5 16:04 jre -> jdk1.5.0_07
-----------------------------------------------------------
III: Install GSI infrastructure and host certificate.
If dCache is going to be accessed via GridFTP and/or SRM. a host certificate is requred. GSI infrastructure and host certificate are required on SRM door, pool nodes, GridFTP door.
Contact the CA responsible for your community for details. The certificate is expected to be installed in /etc/grid-security. In the testbed, dctest03, dctest04 and dctest05 are required to install host certificate.
------------------------------------------------------------------------------
[root@dctest03 grid-security]# chmod 400 /etc/grid-security/hostkey.pem
[root@dctest03 grid-security]# chmod 644 /etc/grid-security/hostcert.pem
------------------------------------------------------------------------------
repeate the same steps on dctest04 and dctest05.
IV. Install Postgres v8.1.4 on PNFS and SRM door node.
1. Download postgres packages.
In this example, we download all packages from ftp://ftp.us.postgresql.org to dctest02 and dctest03.
---------------------------------------------------------------------------------------------
[root@dctest02 tmp_pkg]# mkdir $DCACHE_PKG_PATH/postgres-8.1.4
[root@dctest02 tmp_pkg]# cd $DCACHE_PKG_PATH/postgres-8.1.4
[root@dctest02 postgres-8.1.4]# cd $DCACHE_PKG_PATH/postgres-8.1.4
[root@dctest02 postgres-8.1.4]#wget ftp://ftp.us.postgresql.org/pub/mirrors/postgresql/binary/v8.1.4/linux/rpms/redhat/rhel-as-4/postgresql-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]#wget ftp://ftp.us.postgresql.org/pub/mirrors/postgresql/binary/v8.1.4/linux/rpms/redhat/rhel-as-4/postgresql-contrib-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]#wget ftp://ftp.us.postgresql.org/pub/mirrors/postgresql/binary/v8.1.4/linux/rpms/redhat/rhel-as-4/postgresql-devel-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]#wget ftp://ftp.us.postgresql.org/pub/mirrors/postgresql/binary/v8.1.4/linux/rpms/redhat/rhel-as-4/postgresql-jdbc-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]#wget ftp://ftp.us.postgresql.org/pub/mirrors/postgresql/binary/v8.1.4/linux/rpms/redhat/rhel-as-4/postgresql-libs-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]#wget ftp://ftp.us.postgresql.org/pub/mirrors/postgresql/binary/v8.1.4/linux/rpms/redhat/rhel-as-4/postgresql-pl-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]#wget ftp://ftp.us.postgresql.org/pub/mirrors/postgresql/binary/v8.1.4/linux/rpms/redhat/rhel-as-4/postgresql-python-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]#wget ftp://ftp.us.postgresql.org/pub/mirrors/postgresql/binary/v8.1.4/linux/rpms/redhat/rhel-as-4/postgresql-server-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]#wget ftp://ftp.us.postgresql.org/pub/mirrors/postgresql/binary/v8.1.4/linux/rpms/redhat/rhel-as-4/postgresql-test-8.1.4-1PGDG.i686.rpm
---------------------------------------------------------------------------------------------------
2. Install postgres
------------------------------------------------------------------------------------------
[root@dctest02 postgres-8.1.4]# rpm -Uvh --nodeps postgresql-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]# rpm -Uvh --nodeps postgresql-contrib-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]# rpm -Uvh --nodeps postgresql-devel-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]# rpm -Uvh --nodeps postgresql-jdbc-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]# rpm -Uvh --nodeps postgresql-libs-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]# rpm -Uvh --nodeps postgresql-pl-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]# rpm -Uvh --nodeps postgresql-python-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]# rpm -Uvh --nodeps postgresql-server-8.1.4-1PGDG.i686.rpm
[root@dctest02 postgres-8.1.4]# rpm -Uvh --nodeps postgresql-test-8.1.4-1PGDG.i686.rpm
------------------------------------------------------------------------------------------
3. Check whether postgres is installed correctly.
----------------------------------------------------------------------------------------
[root@dctest02 etc]# rpm -qa | grep postgres
postgresql-8.1.4-1PGDG
postgresql-libs-8.1.4-1PGDG
postgresql-test-8.1.4-1PGDG
postgresql-pl-8.1.4-1PGDG
postgresql-contrib-8.1.4-1PGDG
postgresql-jdbc-8.1.4-1PGDG
postgresql-server-8.1.4-1PGDG
postgresql-devel-8.1.4-1PGDG
postgresql-python-8.1.4-1PGDG
-------------------------------------------------------------------------------------------
Repeat the same steps on dctest03, the SRM node.
V. Installation of the dCache package on all nodes.
/opt/d-cache will be the default dCache installation area. We strongly suggest you to use this default path for the convenience. If you don't have enough space in this directory, please choose a big disk, and make a symbolic link to /opt/d-cache.
1. Get the package:
------------------------------------------------------------------------
[root@dctest01 tmp_pkg]# mkdir $DCACHE_PKG_PATH/dcache-1.6.6.5
[root@dctest01 tmp_pkg]# cd $DCACHE_PKG_PATH/dcache-1.6.6.5
[root@dctest01 dcache-bundle-1.6.6-5]# wget http://www.dcache.org/downloads/releases/dcache-bundle-1.6.6-5.tar
[root@dctest01 dcache-bundle-1.6.6-5]# tar -xvf dcache-bundle-1.6.6-5.tar
---------------------------------------------------------------------------
2. Install the package.
(1) Install dcache-server and dcache-client packages on each node
-----------------------------------------------------------------------------
[root@dctest01 dcache-bundle-1.6.6-5]# rpm -i dcache-server-1.6.6-5.i386.rpm
[root@dctest01 dcache-bundle-1.6.6-5]# rpm -i dcache-client-1.6.6-5.i386.rpm
-----------------------------------------------------------------------------
(2) Install pnfs-postgresql in PNFS node
--------------------------------------------------------------------------------
[root@dctest02 dcache-bundle-1.6.6-5]# rpm -i pnfs-postgresql-3.1.10-3.i386.rpm
--------------------------------------------------------------------------------
VI. Initialize and configure Postgres on PNFS and SRM door node.
The following instructions shall be used to initialize and configure the databases. They need to be executed only following the installation of the database. An upgrade of the dCache code does not require the commands to be executed again. Do the following steps in both pnfs and srm nodes.
1. Login as root, then su postgres
-----------------------------------------
[root@dctest02 root]# su postgres
bash-2.05b$ cd
bash-2.05b$ pwd
/var/lib/pgsql
------------------------------------------
2. Initialize DB
#: /var/lib/pgsql
#initdb -D /var/lib/pgsql/data
-------------------------------------------------
bash-2.05b$ initdb -D /var/lib/pgsql/data
-------------------------------------------------
3. Enable network access(non-local connection) in postgres config file /var/lib/pgsql/data/pg_hba.conf
If the database is used by certain components in admin node, add admin node records in the file. Please refer to dCache book and comments in the pg_hba.conf for more information.
In this example, we add ip address(130.199.185.37) of admin node in the file in PNFS node.
--------------------------------------------------------------------------------
...
# TYPE DATABASE USER CIDR-ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
#dctest02.usatlas.bnl.gov
host all all 130.199.185.37/32 trust
------------------------------------------------------------------------------------
4. Start Postgres (by root account)
#/etc/init.d/postgresql restart
---------------------------------------------------------------------------------------------------
[root@dctest02 postgres8.1.4]# ps -ef | grep post
postgres 22250 1 9 15:56 pts/1 00:00:01 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
postgres 22252 22250 0 15:56 pts/1 00:00:00 postgres: logger process
postgres 22254 22250 0 15:56 pts/1 00:00:00 postgres: writer process
postgres 22255 22250 0 15:56 pts/1 00:00:00 postgres: stats buffer process
postgres 22256 22255 0 15:56 pts/1 00:00:00 postgres: stats collector process
----------------------------------------------------------------------------------------------------
The following steps are exectuted by root account.
5. Create account
(1) Create pnfsserver account
-----------------------------------------------------------------------------------------------------------
[root@dctest02 root] # createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt pnfsserver
------------------------------------------------------------------------------------------------------------
(2) Create srmdcache account
-----------------------------------------------------------------------------------------------------------
[root@dctest02 root] # createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt srmdcache
------------------------------------------------------------------------------------------------------------
6. Create database dcache
-----------------------------------------------------------------------------------------
[root@dctest02 root] # createdb -U srmdcache dcache
-----------------------------------------------------------------------------------
7. The pnfs companion uses the database “companion” to store the pools all files are located on.
------------------------------------------------------------------------------------
[root@dctest02 root] # createdb -U srmdcache companion
[root@dctest02 root] # psql -U srmdcache companion -f /opt/d-cache/etc/psql_install_companion.sql
------------------------------------------------------------------------------------
8. If the resilience feature provided by the replica manager is used, the database “replicas” has to be prepared.
-------------------------------------------------------------------------------------
[root@dctest02 root] # createdb -U srmdcache replicas
[root@dctest02 root] # psql -U srmdcache replicas -f /opt/d-cache/etc/psql_install_replicas.sql
-----------------------------------------------------------------------------------------
9. If the billing information should also be stored in a database (in addition to files) the database “billing” has to be created.
----------------------------------------------------------------------------------------------
[root@dctest02 root] # createdb -U srmdcache billing
---------------------------------------------------------------------------------------------
VII. Install dCache system
1. Install pnfs manager (on PNFS node)
After running rpm -i pnfs-postgresql-3.1.10-3.i386.rpm in step V, /opt/pnfs and /opt/pnfs.3.1.10 are created.
(1) Log in to pnfs node (dctest02)
(2) Copy the template pnfs_config.template to pnfs_config
The content of the pnfs_config template is:
PNFS_INSTALL_DIR = /opt/pnfs
PNFS_ROOT = /pnfs
PNFS_DB = /opt/pnfsdb
PNFS_LOG = /opt/d-cache/log
PNFS_OVERWRITE = no
PNFS_PSQL_USER = pnfsserver
-----------------------------------------------------------------------------------------------
[root@dctest02 root] # cp /opt/pnfs/etc/pnfs_config.template /opt/pnfs/etc/pnfs_config
------------------------------------------------------------------------------------------
(3) Install pnfs manager
-------------------------------------------------------------------
[root@dctest02 config]# /opt/pnfs/install/pnfs-install.sh
PNFS_PSQL_USER = pnfsserver
Checking nfs servers : Ok
Preparing setup : Ok
Creating database admin
Creating database data1
Starting pnfs server ... Ok
Trying to talk to dbserver 0 [1122] ... Ok
Trying to talk to dbserver 1 [1122] ... Ok
Trying to mount 'pnfs' : Ok
Correcting pnfs permissions : Ok
Detecting wormhole target (config) : 0000000000000000000010E0
Digging wormholes : dig-0-ok dig-1-ok Done
Creating database link : Ok
Setting mount permissions to world : Ok
Remarks :
ii) Any host may now mount this pnfs server
mount -o intr,rw,noac,hard,nfsvers=2 :/pnfs /
Installation of PNFS completed - stop PNFS
Stopping Heartbeat .... Ready
Killing pnfsd Done
Killing pmountd Done
Killing dbserver . Done
Removing 8 Clients 0+ 1+ 2+ 3+ 4+ 5+ 6+ 7+
Removing 8 Servers 0+ 1+ 2+ 3+ 4+ 5+ 6+ 7+
Removing main switchboard ... O.K.
-------------------------------------------------------------------
Database will be generated at /opt/pnfsdb. If there's no enough space for /opt/pnfsdb, move it to a large directory, and make a symbolic link.
(4) Start pnfs
#/opt/pnfs/bin/pnfs start|stop
--------------------------------------------------------------------------------
[root@dctest02 root]# /opt/pnfs/bin/pnfs start
--------------------------------------------------------------------------------
(5) Add files under /pnfs/fs/admin/etc/exports to allow mount from admin, SRM node and gridFTP node.
---------------------------------------------------------------------------------
[root@dctest02 root]# cd /pnfs/fs/admin/etc/exports
[root@dctest02 exports]# cat 127.0.0.1
/admin /0/root/fs/admin 0 nooptions
/fs /0/root/fs 0 nooptions
-----------------------------------------------------------------------------------
Use touch and echo to create files with the same content as 127.0.0.1. Name these files with the IP address of admin, srm and GridFtp node. In this example, the IP address of dctest01 is 130.199.185.37.
----------------------------------------------------------------------------------
[root@dctest02 exports]# touch 130.199.185.37
[root@dctest02 exports]# echo "/admin /0/root/fs/admin 0 nooptions" > 130.199.185.37
[root@dctest02 exports]# echo "/fs /0/root/fs 0 nooptions" >> 130.199.185.37
-----------------------------------------------------------------------------------
Repeat the same steps for SRM, gridftp node.
----------------------------------------------------------------------------------
[root@dctest02 exports]#ls -l
total 1
-rw-r--r-- 1 root root 75 Jun 5 15:25 0.0.0.0..0.0.0.0
-rwx------ 1 root bin 92 Jun 5 14:51 127.0.0.1
-rwxr-xr-x 1 root root 92 Jun 2 15:09 130.199.185.37
-rwxr-xr-x 1 root root 92 Jun 2 15:09 130.199.185.38
-rwxr-xr-x 1 root root 92 Jun 2 15:09 130.199.185.39
drwx------ 1 root bin 512 Jun 5 14:51 trusted
----------------------------------------------------------------------------------
2. Install dCache in admin node, dctest01
1. Modify dCacheSetup file in PNFS node.
(1) Login to PNFS node, dctest02, as root.
(2) Create the dCacheSetup file from template
--------------------------------------------------------------
[root@dctest02 config] cp /opt/d-cache/etc/dCacheSetup.template /opt/d-cache/config/dCacheSetup
[root@dctest02 config] vi /opt/d-cache/config/dCacheSetup
-------------------------------------------------------------
(3) Change the following value in dCacheSetup file
serviceLocatorHost=dctest01.usatlas.bnl.gov
java="/usr/java/jre/bin/java "
logArea=/opt/d-cache/log/
2. Modify dCacheSetup file in admin node.
(1) Login to admin node, dctest01, as root.
(2) Create the dCacheSetup file from template
--------------------------------------------------------------
[root@dctest01 config] cp /opt/d-cache/etc/dCacheSetup.template /opt/d-cache/config/dCacheSetup
[root@dctest01 config] vi /opt/d-cache/config/dCacheSetup
-------------------------------------------------------------
(3) Change the following value in dCacheSetup file
serviceLocatorHost=localhost
java="/usr/java/jre/bin/java "
logArea=/opt/d-cache/log/
companionDatabaseHost=dctest02.usatlas.bnl.gov
srmDatabaseHost=dctest03.usatlas.bnl.gov
spaceManagerDatabaseHost=dctest03.usatlas.bnl.gov
pinManagerDatabaseHost=dctest03.usatlas.bnl.gov
replicaManagerDatabaseHost=dctest02.usatlas.bnl.gov
3.Change the node_config file in admin node
(1) Create the node_config file from template
--------------------------------------------------------------
[root@dctest01 etc] cp /opt/d-cache/etc/node_config.template /opt/d-cache/etc/node_config
[root@dctest01 etc] vi /opt/d-cache/etc/node_config
-------------------------------------------------------------
(2) Make the following changes in node_config file.
NODE_TYPE=admin
pnfsManager=no
Add the following line for pnfsServer in the file.
pnfsServer=dctest02.usatlas.bnl.gov
This is the sample of the file in dctest02.usatlas.bnl.gov.
--------------------------------------------------------------------------------
[root@dctest01 pnfs]# more /opt/d-cache/etc/node_config
NODE_TYPE=admin #admin, pool, or door
DCACHE_BASE_DIR=/opt/d-cache
PNFS_ROOT=/pnfs
PNFS_INSTALL_DIR=/opt/pnfs
PNFS_START=yes
PNFS_OVERWRITE=no
POOL_PATH=/opt/d-cache/etc
NUMBER_OF_MOVERS=100
pnfsServer=dctest02.usatlas.bnl.gov
# SERVER_ID=domain.name # defaults to `hostname -d`
ADMIN_NODE=dctest01.usatlas.bnl.gov # only needed for GridFTP door which is not on the admin node
# ---- Services to be started on this node
# The following services are only started on this node
# if the corresponding parameter is set to 'yes'.
# Exeption: The PnfsManager is started on the admin node
# if the parameter is not specified.
#
GSIDCAP=no
GRIDFTP=no
SRM=no
# ---- Start the Replica Manager on this node.
# The variable 'replicaManager' in config/dCacheSetup has to be set
# to 'yes' on every node of the dCache instance, if the replica manager
# is started with the following variable
# Make sure that there is only one replica manager running in a dCache
# instance.
#
#replicaManager=no # default: no
# ---- Start the pnfs Manager on this node.
# With this variable, it is possible to install the pnfs server and
# pnfs manager on a separate node and not on the admin node.
# Make sure that there is only one pnfs manager running in a dCache
# instance.
#
pnfsManager=no # default: 'yes' on 'admin' node otherwise 'no'
# ---- Start the info provider on this node.
# With this variable, it is possible to install the info provider
# on a separate node and not on the admin node.
---------------------------------------------------------------------------------------
4. Run install script in dctest01
--------------------------------------------------------------------------------------
[root@dctest01 etc]/opt/d-cache/install/install.sh
--------------------------------------------------------------------------------------
5. Writing wormhole information in PNFS node.
Since PNFS is not in admin node, the wormhole and tag information is not created when run the install.sh script in admin node.
(1) Login to pnfs node, dctest02.usatlas.bnl.gov, as root
(2) Make sure pnfs server is started.
(3) Create a script called pnfs_write.sh under /opt/d-cache/install.
Make sure use the correct value for server_ID. It should be the value of "hostname -d" of the pnfs node. In this example, it is "usatlas.bnl.gov".
Also please change the value of dcapHostname (the hostname of dcap door) in your enviroment. In this example, it is dctest01.usatlas.bnl.gov.
--------------------------------------------------------------------------
[[root@dctest02 install]# more pnfs_write.sh
#!/bin/sh
PNFS_ROOT=/pnfs
fqHostname=`hostname`
dcapHostname="dctest01.usatlas.bnl.gov"
SERVER_ID="usatlas.bnl.gov"
cd $PNFS_ROOT/fs
serverRoot=`cat ".(id)(usr)"`
# cat /etc/resolv.conf| grep search | awk '{ print($2) }' >./serverId
# SID="`cat /etc/resolv.conf| grep search | awk '{ print($2) }'`"
# if [ -z $SID ]; then
# SID="`cat /etc/resolv.conf| grep domain | awk '{ print($2) }'`"
# fi
# Writing Wormhole information
#
cd $PNFS_ROOT/fs/admin/etc/config
echo "${fqHostname}" > ./serverName
echo "${SERVER_ID}" >./serverId
echo "$serverRoot ." > ./serverRoot
touch ".(fset)(serverName)(io)(on)"
touch ".(fset)(serverId)(io)(on)"
touch ".(fset)(serverRoot)(io)(on)"
echo "${fqHostname}" > ./serverName
echo "${SERVER_ID}" >./serverId
echo "$serverRoot ." > ./serverRoot
mkdir -p dCache
cd dCache
echo "${dcapHostname}:22125" > ./dcache.conf
touch ".(fset)(dcache.conf)(io)(on)"
echo "${dcapHostname}:22125" > ./dcache.conf
# Configure directory tags
#
cd $PNFS_ROOT/fs/usr/data
echo "StoreName myStore" > ".(tag)(OSMTemplate)"
echo STRING > ".(tag)(sGroup)"
---------------------------------------------------------------------------
6. Modify dcache start script in admin node.
(1) Login to admin node as root
(2) Comment PNFS related stuff in the file /opt/d-cache/bin/dcache-core
(3) Comment start script for utility (start in SRM node)
The following is what we did in the example.
---------------------------------------------------------------------------
case "$1" in
start)
...
start)
if [ "`printConfig NODE_TYPE`" = admin ] ; then
$ourHome/jobs/lm ${logParam} start
$ourHome/jobs/dCache ${logParam} start
$ourHome/jobs/dir ${logParam} start
$ourHome/jobs/door ${logParam} start
$ourHome/jobs/adminDoor ${logParam} start
$ourHome/jobs/httpd ${logParam} start
# #$ourHome/jobs/utility ${logParam} start
fi
# if [ -z "`printConfig PnfsManager`" -a "`printConfig NODE_TYPE`" = admin -o "`printConfig PnfsManager`" = yes ] ; then
# $ourHome/jobs/pnfs ${logParam} start
# fi
---------------------------------------------------------------------------
7. Modify dcache start script in pnfs node.
(1) Login to pnfs node as root
(2) Modify file /opt/d-cache/bin/dcache-core, make sure only pnfs Manager is started.
----------------------------------------------------------------
echo "Starting dcache services: "
#if [ "`printConfig NODE_TYPE`" = admin ] ; then
# $ourHome/jobs/lm ${logParam} start
# $ourHome/jobs/dCache ${logParam} start
# $ourHome/jobs/dir ${logParam} start
# $ourHome/jobs/door ${logParam} start
# $ourHome/jobs/adminDoor ${logParam} start
# $ourHome/jobs/httpd ${logParam} start
# $ourHome/jobs/utility ${logParam} start
#fi
if [ -z "`printConfig PnfsManager`" -a "`printConfig NODE_TYPE`" = admin -o "`printConfig PnfsManager`" = yes ] ; then
$ourHome/jobs/pnfs ${logParam} start
fi
if [ -z "`printConfig InfoProvider`" -a "`printConfig NODE_TYPE`" = admin -o "`printConfig InfoProvider`" = yes ] ; then
$ourHome/jobs/infoProvider ${logParam} start
fi
--------------------------------------------------------------------
8. Start dCache in admin node. Services in admin node have to start first before starting PNFS Manager in pnfs node.
(1) start service
----------------------------------------------------------------
[root@dctest01 bin]# ./dcache-core start
Starting dcache services:
Starting lmDomain 6 5 4 3 2 1 0 Done (pid=31960)
Starting dCacheDomain 6 5 4 3 2 1 0 Done (pid=32022)
Starting dirDomain 6 5 4 3 2 1 0 Done (pid=32098)
Starting doorDomain 6 5 4 3 2 1 0 Done (pid=32170)
Starting adminDoorDomain 6 5 4 3 2 1 0 Done (pid=32245)
Starting httpdDomain 6 5 4 3 2 1 0 Done (pid=32323)
Starting infoProviderDomain 6 5 4 3 2 1 0 Done (pid=32497)
-------------------------------------------------------
(2) Verify the services by list listening port
--------------------------------------------------------------
[root@dctest01 pnfs]# netstat -lntp | grep java
tcp 0 0 :::32810 :::* LISTEN 3711/java
tcp 0 0 :::22125 :::* LISTEN 3858/java
tcp 0 0 :::22223 :::* LISTEN 3933/java
tcp 0 0 :::2288 :::* LISTEN 19801/java
--------------------------------------------------------
9. Start PNFS Manager in pnfs node.
------------------------------------------------------------------
[root@dctest02 bin]# ./dcache-core start
Starting dcache services:
Starting pnfsDomain 6 5 4 3 2 1 0 Done (pid=32028)
-----------------------------------------------------------------
10. Verify PNFS Manager is started and admin node is correctly mounted.
-------------------------------------------------------------------
[root@dctest01 pnfs]# ls -l
total 5
drwxrwxrwx 1 root root 512 Jun 5 15:44 fs
lrwxrwxrwx 1 root root 20 Jul 19 16:07 usatlas.bnl.gov -> /direct/usatlas+pnfs
------------------------------------------------------------------
VIII. Install SRM server in SRM node, dctest03.usatlas.bnl.gov
1. Login to SRM node as root.
2. Modify node_config file
(1) Create the node_config file from template
--------------------------------------------------------------
[root@dctest03 etc] cp /opt/d-cache/etc/node_config.template /opt/d-cache/etc/node_config
[root@dctest03 etc] vi /opt/d-cache/etc/node_config
-------------------------------------------------------------
(2) Make the following changes in node_config file.
NODE_TYPE=door
ADMIN_NODE=dctest01.usatlas.bnl.gov
SRM=yes
4. Modify dCacheSetup file
(1) Create the dCacheSetup file from template
--------------------------------------------------------------
[root@dctest03 config] cp /opt/d-cache/etc/dCacheSetup.template /opt/d-cache/config/dCacheSetup
[root@dctest03 config] vi /opt/d-cache/config/dCacheSetup
-------------------------------------------------------------
(2) Change the following value in dCacheSetup file
serviceLocatorHost=dctest01.usatlas.bnl.gov
java="/usr/java/jre/bin/java "
logArea=/opt/d-cache/log/
companionDatabaseHost=dctest02.usatlas.bnl.gov
srmDatabaseHost=localhost
spaceManagerDatabaseHost=localhost
pinManagerDatabaseHost=localhost
replicaManagerDatabaseHost=dctest02.usatlas.bnl.gov
5. Run installation script
---------------------------------------------------------------------
[root@dctest03 install]# ./install.sh
[INFO] No 'SERVER_ID' set in 'node_config'. Using SERVER_ID=usatlas.bnl.gov.
[WARN] /opt/d-cache/etc/pool_path does not exist. No pools will be configured
----------------------------------------------------------------------
6. Modify dCache start script
Add utility start before srmDomain start.
----------------------------------------------------------------------
if [ "`printConfig SRM`" = yes ] ; then
$ourHome/jobs/utility ${logParam} start
$ourHome/jobs/srm -domain=srm-${shortHostname}Domain ${logParam} start
fi
--------------------------------------------------------------------
7. Start SRM server
----------------------------------------------------------------------
[root@dctest03 install]# /opt/d-cache/bin/dcache-core start
Starting dcache services:
Starting utilityDomain 6 5 4 3 2 1 0 Done (pid=9609)
Starting srm-dctest04Domain 6 5 4 3 2 1 0 Done (pid=9695)
--------------------------------------------------------------------------
8. Verify the service is started by listing listening port.
-----------------------------------------------------------------------
[root@dctest03 ~]# netstat -lntp | grep java
tcp 0 0 :::8443 :::* LISTEN 18862/java
----------------------------------------------------------------------
IX. Install door node
1. Login to door node, dctest04.usatlas.bnl.gov as root.
2. Modify node_config file
(1) Create the node_config file from template
--------------------------------------------------------------
[root@dctest04 etc] cp /opt/d-cache/etc/node_config.template /opt/d-cache/etc/node_config
[root@dctest04 etc] vi /opt/d-cache/etc/node_config
-------------------------------------------------------------
(2) Make the following changes in node_config file.
NODE_TYPE=door
ADMIN_NODE=dctest01.usatlas.bnl.gov
GRIDFTP=yes
4. Modify dCacheSetup file
(1) Create the dCacheSetup file from template
--------------------------------------------------------------
[root@dctest04 config] cp /opt/d-cache/etc/dCacheSetup.template /opt/d-cache/config/dCacheSetup
[root@dctest04 config] vi /opt/d-cache/config/dCacheSetup
-------------------------------------------------------------
(2) Change the following value in dCacheSetup file
serviceLocatorHost=dctest01.usatlas.bnl.gov
java="/usr/java/jre/bin/java "
logArea=/opt/d-cache/log/
5. Run install script
-----------------------------------------------------
[root@dctest04 install] /opt/d-cache/install/install.sh
-----------------------------------------------------
Make sure the door node is mounted correctly as the following. ftpBase should be created.
---------------------------------------------------------------
[root@dctest04 pnfs]# ls -l
total 9
drwxrwxrwx 1 root root 512 Jun 5 15:44 fs
lrwxrwxrwx 1 root root 21 Jul 20 22:05 ftpBase -> /pnfs/usatlas.bnl.gov
lrwxrwxrwx 1 root root 6 Jul 20 21:55 usatlas.bnl.gov -> fs/usr
-------------------------------------------------------------
6.Start dCache GridFTP door
/opt/d-cache/bin/dcache-core start
7. Verify gridftp service by check the listneing ports
--------------------------------------------------------------
[root@dctest04 java]# netstat -lntp | grep java
tcp 0 0 :::2811 :::* LISTEN 10746/java
-------------------------------------------------------------
X. Install pool node
1. Login to pool node, dctest05.usatlas.bnl.gov as root.
2. Modify node_config file
(1) Create the node_config file from template
--------------------------------------------------------------
[root@dctest05 etc] cp /opt/d-cache/etc/node_config.template /opt/d-cache/etc/node_config
[root@dctest05 etc] vi /opt/d-cache/etc/node_config
-------------------------------------------------------------
(2) Make the following changes in node_config file.
NODE_TYPE=pool
ADMIN_NODE=dctest01.usatlas.bnl.gov
4. Modify dCacheSetup file
(1) Create the dCacheSetup file from template
--------------------------------------------------------------
[root@dctest05 config] cp /opt/d-cache/etc/dCacheSetup.template /opt/d-cache/config/dCacheSetup
[root@dctest05 config] vi /opt/d-cache/config/dCacheSetup
-------------------------------------------------------------
(2) Change the following value in dCacheSetup file
serviceLocatorHost=dctest01.usatlas.bnl.gov
java="/usr/java/jre/bin/java "
logArea=/opt/d-cache/log/
5. Create and Modify pool_path file
Create the pool_path file from template
The template contains pool parameters (path, size, etc)
The format of the pool_path file is (3 columns)
/path/to/pool1 size[GB] "overwrite if exists (yes/no)"
[Note: GB means 1024^3; space for inodes etc. is not accounted for]
--------------------------------------------------------------
[root@dctest05 config] cp /opt/d-cache/etc/pool_path.template /opt/d-cache/etc/pool_path
[root@dctest05 config] vi /opt/d-cache/etc/pool_path
/data/dcache_pool_1 5 no
-------------------------------------------------------------
6.Run install script
-----------------------------------------------------
[root@dctest05 install] /opt/d-cache/install/install.sh
-----------------------------------------------------
7.Start dCache pool
/opt/d-cache/bin/dcache-pool start
8. Verify pool is created.
------------------------------------------------------------------
[root@dctest05 dcache_pool_1] pwd
/data/dcache_pool_1/pool
[root@dctest06 pool]# ls -l
total 40
drwxr-xr-x 2 root root 4096 Jul 14 19:09 control
drwxr-xr-x 2 root root 4096 Jun 6 17:22 data
-rw-r--r-- 1 root root 1119 Jun 2 19:04 setup
-rw-r--r-- 1 root root 1120 Jun 2 19:04 setup.orig
-rw-r--r-- 1 root root 1118 Jun 2 19:04 setup.temp
-------------------------------------------------------------------
XI. Post-Installation
1. Authentication file: dcache.kpwd
For authorization of grid users the file /opt/d-cache/etc/dcache.kpwd is needed in SRM and GridFTP door. Note that it may be generated from the standard /etc/grid-security/grid-mapfile with the tool grid-mapfile2dcache-kpwd which is distributed with the LCG software.
2. The Web Interface for Monitoring dCache
dCache provides a web based interface showing the status of system in terms of Cell Services, Pool (Space) Usage, Pool Request Queues, Pools, Actions Log, Transfer Knowledge, Restore Queue. The contents of the web interface are self-explanatory and are the primary source for most monitoring and trouble-shooting tasks.
Default monitoring interface: http://:2288/
In this example, it is http://dctest01.usatlas.bnl.gov:2288/
The default installation doesn't include several stuff, like cleaner, also, it doesn't show some important cells. We can change httpd.batch to expert those cells and make them available on the web. Please refer to dCache book for more details.