In this post we are going to learn how to configure GPFS cluster configuration in AIX environment.

So we are going to configure the GPFS cluster with two nodes setup. If you want to learn GPFS full concepts you can refer IBM site.

Setup and Preparation:

Operating System Setup:

  • To Build two aix nodes with same OS level (AIX 6.1.TL9-Spx or 7.1TL4-spX) along with latest updates, and configure the Host name and IP address.
  • Verify the rootvg disk path status.
  • Mirror rootvg and set bosboot and bootlist.

Storage Setup:

  • Need shared storage LUNS between both the nodes for GPFS File system configuration.
  • Minimum Requirements for NSD : Two Disk with any size
  • Minimum Requirements for Tiebreakr Disk : One disk with size of 512 MB or 1 GB.

Network Setup:

  • In our setup we have two Nodes cluster. One is Primary Node another one is Secondary node but in our cause both are quorum nodes.
  • We need Two Private IPv4 address for GPFS cluster intercommunication Purpose which is non rout able IP.
  • And we need two Admin IP address for Admin access Purpose, through this IP’s only admin can able to access the servers.

Make both the nodes GPFS Nodes Host Entry

NODE1+NODE2:# cat /etc/hosts
#######NODE IP’s############
192.168.1.50 NODE1
192.168.1.51 NODE2
#########GPFS-NODES#########
10.10.1.15 NODE1-GPFS
10.10.1.16 NODE2-GPFS

Create Passowrd less SSH between cluster nodes for cluster communication purpose.

Take Backup of sshd_config file

NODE1+NODE2:# cp -p /etc/ssh/sshd_config /etc/ssh/sshd_config-bkp

For enabling the password less ssh, Add “PermitRootLogin without-password” lines into sshd_config file.

NODE1+NODE2:# cat -n /etc/ssh/sshd_config | grep -i PermitRootLogin
#The setting of “PermitRootLogin without-password”
PermitRootLogin without-password

Restart the sshd services, the only changes will effect.

stopsrc -s sshd;startsrc -s sshd

To Generate Public key as per mentioned commands and copy the authorized_keys into both the nodes.

NODE1+NODE2:# ssh-keygen -t rsa -N “” -f $HOME/.ssh/id_rsa
NODE1+NODE2:# cp $HOME/.ssh/id_rsa.pub $HOME/.ssh/authorized_keys
NODE1:# scp $HOME/.ssh/id_rsa.pub root@192.168.1.51:/tmp/id_rsa.pub
NODE2:# scp $HOME/.ssh/id_rsa.pub root@192.168.1.50:/tmp/id_rsa.pub
NODE1+NODE2:# cat /tmp/id_rsa.pub >> $HOME/.ssh/authorized_keys
NODE1+NODE2:# cat $HOME/.ssh/authorized_keys
NODE1+NODE2:# ls -ld /home/root/.ssh/authorized_keys
-rw——- 1 root system 784 Jul 24 11:44 /home/root/.ssh/authorized_keys
NODE1+NODE2:# ls -ld /home/root
drwx—— 5 root system 4096 Jul 24 14:52 /home/root
NODE1+NODE2:# ls -ld /home/root/.ssh
drwxr-xr-x 2 root system 256 Jul 24 11:42 /home/root/.ssh

To set the Storage Shared Luns Disks Attributes on both the Nodes.

NODE1+NODE2:# ldev -Cc disk | grep -i MPIO
NODE1+NODE2:# chdev -l hdisk<n> -a algorithm=round_robin -a queue_depth=32 -a reserve_policy=no_reserve
NODE1+NODE2:# lsattr -El hdisk<n> -a queue_depth -a algorithm -a reserve_policy

GPFS base File sets software Installation on both the nodes.

NODE1+NODE2:# mount 192.168.1.100:/Softwares/GPFS-3.5 /mnt
NODE1+NODE2:# cd /mnt/IBM-Spectrum-Scale/GPFS-3.5/GPFS3.5_base
NODE1+NODE2:# smitty install_all

After software installation take both the nodes reboot.

To update GPFS base file sets into latest level :

NODE1+NODE2:# cd /mnt/IBM-Spectrum-Scale/GPFS-3.5/GPFS-3.5.0.28
NODE1+NODE2:# install_all_updates -Yd . all
NODE1+NODE2:# lslpp -l | grep -i gpfs
gpfs.base 3.5.0.28 APPLIED GPFS File Manager
gpfs.gnr 3.5.0.3 COMMITTED GPFS Native RAID
gpfs.msg.en_US 3.5.0.23 APPLIED GPFS Server Messages – U.S.
gpfs.base 3.5.0.28 APPLIED GPFS File Manager
gpfs.docs.data 3.5.0.4 APPLIED GPFS Server Manpages and

To Needs do be export GPFS Path in .profile file.

NODE1+NODE2:# cp -p /home/root/.profile /home/root/.profile-bkp
NODE1+NODE2:# cat>>/home/root/.profile
export PATH=$PATH:/usr/lpp/mmfs/bin

To create stanzha file for GPFS Cluster configuration purpose.

NODE1+NODE2:# mkdir /gpfs/conf

Prerequisites has been completed and Start the GPFS cluster configuration.

Create a stanza file along with below mentioned details.

NODE1:# cat >> /gpfs/conf/NodeDesc
NODE1-GPFS:manager-quorum
NODE2-GPFS:manager-quorum

On Node one Start the cluster configuration.

NODE1:# /usr/lpp/mmfs/bin/mmcrcluster -N /gpfs/conf/NodeDesc -p NODE1-GPFS -s NODE2-GPFS -r /usr/bin/ssh -R /usr/bin/scp -C GPFSTESTCLUSTER -A
Mon Dec 19 00:39:51 IST 2016: mmcrcluster: Processing node NODE1-GPFS
Mon Dec 19 00:39:54 IST 2016: mmcrcluster: Processing node NODE2-GPFS
mmcrcluster: Command successfully completed
mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
Use the mmchlicense command to designate licenses as needed.
mmcrcluster: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

Once cluster created to  verify the cluster details.

NODE1:# mmlscluster
===============================================================================
| Warning: |
| This cluster contains nodes that do not have a proper GPFS license |
| designation. This violates the terms of the GPFS licensing agreement. |
| Use the mmchlicense command and assign the appropriate GPFS licenses |
| to each of the nodes in the cluster. For more information about GPFS |
| license designation, see the Concepts, Planning, and Installation Guide. |
===============================================================================
GPFS cluster information
========================
GPFS cluster name: GPFSTESTCLUSTER.NODE1-GPFS
GPFS cluster id: 835838342801579773
GPFS UID domain: GPFSTESTCLUSTER.NODE1-GPFS
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
GPFS cluster configuration servers:
———————————–
Primary server: NODE1-GPFS
Secondary server: NODE2-GPFS
Node Daemon node name IP address Admin node name Designation
——————————————————————
1 NODE1-GPFS 10.10.1.15 NODE1-GPFS quorum-manager
2 NODE2-GPFS 10.10.1.16 NODE2-GPFS quorum-manager

To remove the cluster warning messages do accept GPFS Cluster License Agreement.

NODE1:# mmchlicense server –accept -N NODE1-GPFS,NODE2-GPFS
The following nodes will be designated as possessing GPFS server licenses:
NODE1-GPFS
NODE2-GPFS
mmchlicense: Command successfully completed
mmchlicense: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

To verify the cluster license accepted details,

NODE1:# mmlscluster
GPFS cluster information
========================
GPFS cluster name: GPFSTESTCLUSTER.NODE1-GPFS
GPFS cluster id: 835838342801579773
GPFS UID domain: GPFSTESTCLUSTER.NODE1-GPFS
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
GPFS cluster configuration servers:
———————————–
Primary server: NODE1-GPFS
Secondary server: NODE2-GPFS
Node Daemon node name IP address Admin node name Designation
——————————————————————
1 NODE1-GPFS 10.10.1.15 NODE1-GPFS quorum-manager
2 NODE2-GPFS 10.10.1.16 NODE2-GPFS quorum-manager

To check cluster configurations

NODE1:# mmlsconfig
Configuration data for cluster GPFSTESTCLUSTER.NODE1-GPFS:
———————————————————-
myNodeConfigNumber 1
clusterName GPFSTESTCLUSTER.NODE1-GPFS
clusterId 835838342801579773
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.5.0.11
adminMode central
File systems in cluster GPFSTESTCLUSTER.NODE1-GPFS:
—————————————————
(none)

To change cluster Admin Mode Policy for central to allToAll.

NODE1:# mmchconfig adminMode=allToAll
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
NODE1:# mmlsconfig
Configuration data for cluster GPFSTESTCLUSTER.NODE1-GPFS:
———————————————————-
myNodeConfigNumber 1
clusterName GPFSTESTCLUSTER.NODE1-GPFS
clusterId 835838342801579773
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.5.0.11
adminMode allToAll
File systems in cluster GPFSTESTCLUSTER.NODE1-GPFS:
—————————————————
(none)

To start the GPFS cluster on ALL nodes, and get all cluster node status.

NODE1:# mmgetstate -aLs
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
————————————————————————————
1 NODE1-GPFS 0 0 2 down quorum node
2 NODE2-GPFS 0 0 2 down quorum node
Summary information
———————
mmgetstate: Information cannot be displayed. Either none of the
nodes in the cluster are reachable, or GPFS is down on all of the nodes.
To get current node cluster status
NODE1:# mmgetstate
Node number Node name GPFS state
——————————————
1 NODE1-GPFS down

To start the cluster all the nodes :

NODE1:# mmstartup -a
Mon Dec 19 00:56:26 IST 2016: mmstartup: Starting GPFS …
NODE1:# mmgetstate -aLs
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
————————————————————————————
1 NODE1-GPFS 2 2 2 active quorum node
2 NODE2-GPFS 2 2 2 active quorum node
Summary information
———————
Number of nodes defined in the cluster: 2
Number of local nodes active in the cluster: 2
Number of remote nodes joined in this cluster: 0
Number of quorum nodes defined in the cluster: 2
Number of quorum nodes active in the cluster: 2
Quorum = 2, Quorum achieved

To view latest cluster event status.

NODE1+NODE2:# tail -f /var/adm/ras/mmfs.log.latest

To Configuring Network Shared Disks (NSD Disk)

NODE1:# lsdev -Cc disk | grep -i MPIO
hdisk3 Available 35-T1-01 MPIO IBM 2107 FC Disk
hdisk4 Available 35-T1-01 MPIO IBM 2107 FC Disk
hdisk5 Available 35-T1-01 MPIO IBM 2107 FC Disk
NODE1:# lspv | grep -wE “hdisk3|hdisk4|hdisk5”
hdisk3 none None
hdisk4 none None
hdisk5 none None

To generate PVID for all the GPFS cluster shared disks.

NODE1+NODE2:# for i in 3 4 5; do chdev -l hdisk$i -a pv=yes ; done
(or)
NODE1+NODE2:# chdev -l hdisk3 -a pv=yes
NODE1+NODE2:# chdev -l hdisk4 -a pv=yes
NODE1+NODE2:# chdev -l hdisk5 -a pv=yes

To verify the shared disks PVID information’s.

NODE1+NODE2:# lspv | grep -wE “hdisk3|hdisk4|hdisk5”
hdisk3 00f65ee0136cd9f2 None
hdisk4 00f65ee0136cda39 None
hdisk5 00f65ee0136cda65 None

To format the shared disk before adding the disk into cluster.

NODE1+NODE2:# for i in 3 4 5 ; do dd if=/dev/zero of=/dev/hdisk$i bs=4096 count=5000; done
(or)
NODE1+NODE2:# dd if=/dev/zero of=/dev/hdisk3 bs=4096 count=5000
NODE1+NODE2:# dd if=/dev/zero of=/dev/hdisk4 bs=4096 count=5000
NODE1+NODE2:# dd if=/dev/zero of=/dev/hdisk5 bs=4096 count=5000

To Configuring Network Shared Disks

Create one NSD disks stanza file to create NSD disks.

NODE1:# cat>>/gpfs/conf/DiskDesc
hdisk3:NODE1-GPFS::dataAndMetadata::nsd1::
hdisk4:NODE1-GPFS::dataAndMetadata::nsd2::
hdisk5:NODE1-GPFS::dataAndMetadata::Tiebreak1::
NODE1:# mmcrnsd -F /gpfs/conf/DiskDesc
mmcrnsd: Processing disk hdisk3
mmcrnsd: Processing disk hdisk4
mmcrnsd: Processing disk hdisk5
mmcrnsd: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

verify the created NSD disk information’s.

NODE1:# lspv | grep -wE “hdisk3|hdisk4|hdisk5”
hdisk3 00f65ee0136cd9f2 nsd1
hdisk4 00f65ee0136cda39 nsd2
hdisk5 00f65ee0136cda65 Tiebreak1
NODE1:# mmlsnsd
File system Disk name NSD servers
—————————————————————————
(free disk) Tiebreak1 NODE1-GPFS
(free disk) nsd1 NODE1-GPFS
(free disk) nsd2 NODE1-GPFS

To assign NSD disks  into remaining available nodes in the cluster.

NODE1:# cat /gpfs/conf/DiskDesc | grep -vE “#”
nsd1:::dataAndMetadata:-1::system
nsd2:::dataAndMetadata:-1::system
Tiebreak1:::dataAndMetadata:-1::system
NODE1:# cat /gpfs/conf/DiskDesc | grep -vE “#”>>/gpfs/conf/DiskDesctiensd
NODE1:# cat /gpfs/conf/DiskDesctiensd
nsd1:::dataAndMetadata:-1::system
nsd2:::dataAndMetadata:-1::system
Tiebreak1:::dataAndMetadata:-1::system
NODE1:# cat >>/gpfs/conf/NSDServerAssign
nsd1:NODE1-GPFS,NODE2-GPFS
nsd2:NODE1-GPFS,NODE2-GPFS
Tiebreak1:NODE1-GPFS,NODE2-GPFS
NODE1:# mmlsnsd
File system Disk name NSD servers
—————————————————————————
(free disk) Tiebreak1 NODE1-GPFS
(free disk) nsd1 NODE1-GPFS
(free disk) nsd2 NODE1-GPFS
NODE1:# mmchnsd -F /gpfs/conf/NSDServerAssign
mmchnsd: Processing disk nsd1
mmchnsd: Processing disk nsd2
mmchnsd: Processing disk Tiebreak1
mmchnsd: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

To verify the NSD disks and nodes details.

NODE1:# mmlsnsd
File system Disk name NSD servers
—————————————————————————
(free disk) Tiebreak1 NODE1-GPFS,NODE2-GPFS
(free disk) nsd1 NODE1-GPFS,NODE2-GPFS
(free disk) nsd2 NODE1-GPFS,NODE2-GPFS

To add Tiebreak into the cluster for QUORUM Purpose
First we have to stop the cluster to assign Tiebreak into the cluster

NODE1:# mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
————————————————————————————
1 NODE1-GPFS 2 2 2 active quorum node
2 NODE2-GPFS 2 2 2 active quorum node
NODE1:# mmshutdown -a
Mon Dec 19 01:30:39 IST 2016: mmshutdown: Starting force unmount of GPFS file systems
Mon Dec 19 01:30:44 IST 2016: mmshutdown: Shutting down GPFS daemons
NODE2-GPFS: Shutting down!
NODE1-GPFS: Shutting down!
NODE2-GPFS: ‘shutdown’ command about to kill process 9175142
NODE1-GPFS: ‘shutdown’ command about to kill process 7340158
Mon Dec 19 01:30:52 IST 2016: mmshutdown: Finished

To verify the cluster down or not.

NODE1:# mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
————————————————————————————
1 NODE1-GPFS 0 0 2 down quorum node
2 NODE2-GPFS 0 0 2 down quorum node

To add the Tiebreak disk into the cluster this will be used for cluster High Availability (quorum Availability Purpose) .

NODE1:# mmchconfig tiebreakerDisks=Tiebreak1
Verifying GPFS is stopped on all nodes …
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

To verify the Tiebreak disk information’s in cluster setup.

NODE1:# mmlsconfig
Configuration data for cluster GPFSTESTCLUSTER.NODE1-GPFS:
———————————————————-
myNodeConfigNumber 1
clusterName GPFSTESTCLUSTER.NODE1-GPFS
clusterId 835838342801579773
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.5.0.11
tiebreakerDisks Tiebreak1
adminMode allToAll
File systems in cluster GPFSTESTCLUSTER.NODE1-GPFS:
—————————————————

To start the cluster with tiebreaker disks.

NODE1:# mmstartup -a
Mon Dec 19 01:32:43 IST 2016: mmstartup: Starting GPFS …
NODE1:# mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
————————————————————————————
1 NODE1-GPFS 1* 2 2 active quorum node
2 NODE2-GPFS 1* 2 2 active quorum node
————————————————————————————————————–
To get NSD disk extended output

To get NSD disk extended output

NODE1:# mmlsnsd -X
Disk name NSD volume ID Device Devtype Node name Remarks
—————————————————————————————————
Tiebreak1 AC1E32F85856E7D9 /dev/hdisk5 hdisk NODE1-GPFS server node
Tiebreak1 AC1E32F85856E7D9 /dev/hdisk5 hdisk NODE2-GPFS server node
nsd1 AC1E32F85856E7D7 /dev/hdisk3 hdisk NODE1-GPFS server node
nsd1 AC1E32F85856E7D7 /dev/hdisk3 hdisk NODE2-GPFS server node
nsd2 AC1E32F85856E7D8 /dev/hdisk4 hdisk NODE1-GPFS server node
nsd2 AC1E32F85856E7D8 /dev/hdisk4 hdisk NODE2-GPFS server node
NODE1:# mmlsnsd
File system Disk name NSD servers
—————————————————————————
(free disk) Tiebreak1 NODE1-GPFS,NODE2-GPFS
(free disk) nsd1 NODE1-GPFS,NODE2-GPFS
(free disk) nsd2 NODE1-GPFS,NODE2-GPFS

To create GPFS File system sp first we have make one stanza file.

NODE1:# cat /gpfs/conf/NSDfilesystemcreation
nsd1:NODE1-GPFS,NODE2-GPFS
NODE1:# mmcrfs /dev/testgpfs -F /gpfs/conf/NSDfilesystemcreation -T /testgpfs -A yes
The following disks of testgpfs will be formatted on node NODE1:
nsd1: size 142606336 KB
nsd2: size 142606336 KB
Tiebreak1: size 1048576 KB
Formatting file system …
Disks up to size 1.2 TB can be added to storage pool system.
Creating Inode File
Creating Allocation Maps
Creating Log Files
Clearing Inode Allocation Map
Clearing Block Allocation Map
Formatting Allocation Map for storage pool system
Completed creation of file system /dev/testgpfs.
mmcrfs: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
To mount created file system into all the Nodes

To mount created file system into all the Nodes

NODE1:# mmmount /testgpfs -a
Mon Dec 19 01:49:12 IST 2016: mmmount: Mounting file systems …
NODE1:# mmlsnsd
File system Disk name NSD servers
—————————————————————————
testgpfs nsd1 NODE1-GPFS,NODE2-GPFS
testgpfs nsd2 NODE1-GPFS,NODE2-GPFS
testgpfs Tiebreak1 NODE1-GPFS,NODE2-GPFS
NODE1:# df -gt /testgpfs
Filesystem GB blocks Used Free %Used Mounted on
/dev/testgpfs 273.00 0.30 272.70 1% /testgpfs
NODE1:# lsfs /testgpfs
Name Nodename Mount Pt VFS Size Options Auto Accounting
/dev/testgpfs – /testgpfs mmfs — rw,mtime,atime,dev=testgpfs no no