Cluster
=======

IBM PowerHA 7.1 heartbeat over SAN
PowerHA Limits
==============

Component Maximum number/other limits
Nodes 32
Resource groups 64
Networks 48
Network interfaces, devices, and labels 256
Cluster resources While 128 is the maximum that clinfo can handle, there can be more in the cluster
Parent-Child dependencies Max of 3 levels
Site 2
Interfaces 7 interfaces per node per network
Application monitors per site 128
Persistent IP alias One per node per network
XD_data networks 4 per cluster
GLVM Modes Synchronous, Asynchronous, non concurrent
GLVM Devices All PVs supported by AIX, no need to be same local and remote
Log Files
=========
File Description
/var/hacmp/adm/cluster.log Generated by cluster scripts and daemons
/var/hacmp/log/hacmp.out Generated by event scripts and utilities
/var/hacmp/adm/history/cluster.mmddyyyy Cluster history files generated daily
/var/hacmp/clcomd/clcomd.log Generated by clcomd daemon
/var/hacmp/clcomd/clcomddiag.log Generated by clcomd daemon, debug information
/var/hacmp/clverify/clverify.log Generated by Cluster Verification utility
/var/hacmp/log/autoverify.log Generated by Auto Verify and Synchronize
/var/hacmp/log/clavan.log Generated by Application Availability Analysis tool
/var/hacmp/log/clinfo.log Generated by client node running clinfo
/var/hacmp/log/cl_testtool.log Generated by the Cluster Test Tool
/var/hacmp/log/clconfigassist.log Generated by Two-Node Cluster Configuration Assistant
/var/hacmp/log/clstrmgr.debug Generated by the clstrmgr daemon
/var/hacmp/log/clstrmgr.debug.long Detail information from the clstrmgr daemon
/var/hacmp/log/clutils.log Generated by cluster utilities and file propagation
/var/hacmp/log/cspoc.log Generated by CSPOC commands
/var/hacmp/log/cspoc.log.remote Detail information from CSPOC commands
/var/hacmp/log/migration.log Generated by cluster migration
/var/hacmp/log/sa.log Generated by Application Discovery
“odmget HACMPlogs” Displays a complete list of HACMP Log Files
/var/ha/log/topsvcs.default Topology Services starup configuration log
/var/ha/log/topsvcs.dd.hhmmss.lang Topology Services start/stop log
/var/ha/log/topsvcs.dd.hhmmss Topology Services activity log
/var/ha/log/nim.topsvcs.IF.clustername NIM hearthbeat activity log for each interface
/var/ha/log/nmDiag.topsvcs.IF.cluster NIM diagnostic log for each interface
/var/ha/log/grpsvcs.default.nodenum_instnum Group Services startup log
/var/ha/log/grpsvcs_nodenum_instnum Group Services activity log

Useful HACMP Commands
=====================
Command Purpose
clstat Displays topology and resource groups status. clinfoES and snmpd must be running.
cldump Displays topology & resource group status and configuration. snmpd must be running.
cldisp Like cldump, but application oriented. snmpd must be running.
cltopinfo (cllsif) Displays topology configuration.
clRGinfo (clfindres) Displays resource group status.
clsshowres Displays resource groups configuration.
clshowsrv Calls lsrsc to display status of:
HACMP subsystems (clshowrsrv -a)
HACMP and RSCT subsystems (clshowsrv -v)
clcycle Rotates selected log files.
clgetactivenodes Displays active nodes. Must specify which nodo to ask (-n node)
clsnap Save HACMP log files and configuration information.
cl_ls_shared_vgs List shared vgs.
cl_lsfs List shared fs.
cllsgrp List the resource groups.
cllsres Show short resource group information.
clRGmove Brings a RG Offline/Online or move it
lssrc -ls clstrmgrES Displays Cluster Services
Installation changes
——————–
The following AIX configuration changes are made:

1. Files modified:

/etc/hosts
/etc/inittab
/etc/rc.net
/etc/services
/etc/snmpd.conf
/etc/snmpd.peers
/etc/syslog.conf
/etc/trcfmt
/var/spool/cron/crontab/root

2. The hacmp group is added.

3. Also, using cluster configuration and verification, the file /etc/hosts can be changed by adding or modifying entries.

4. The following network options are set to “1” by RSCT topsvcs startup:
– nonlocsrcroute
– ipsrcrouterecv
– ipsrcroutesend
– ipsrcrouteforward
– ip6forwarding

5. The verification utility ensures that the value of each network option is consistent across all cluster nodes for the following settings:
– tcp_pmtu_discover
– udp_pmtu_discover
– ipignoreredirects
– routerevalidate

The cluster communications daemon
———————————
With the introduction of clcomdES, there is no need for an /.rhosts file to be configured. The cluster communications daemon is started by inittab, with the entry being created by the installation of PowerHA. The daemon is controlled by the system resource controller, so startsrc, stopsrc and refresh work. In particular, refresh is used to re-read /usr/es/sbin/cluster/etc/rhosts and moving the log files. The cluster communication daemon uses port 6191.

Resource group components:
————————-
Service IP Label
Volume Group
Filesystem
Application Server
NFS mounts
NFS exports

Resource group Startup options:
——————————
Online on home node only.
Online on first available node.
Online on all available nodes.
Online using distribution policy.

Resource group Fallover options:
——————————-
Fall over to next priority node in list:
Fallover using dynamic node priority:
Bring offline (on error only)

Resource group Fallback options:
——————————-
Fall back to higher priority node in list
Never fall back

Resource group attributes
————————-
Settling time
Delayed fallback timers
Distribution policy
Dynamic node priorities
Resource group processing order
Priority override location
Resource group dependencies – parent / child
Resource group dependencies – location

Resource Groups operations
————————–
Bring a resource group offline
Bring a resource group online
Move a resource group to another node/site
Suspend/resume application monitoring