HOME

manintheit.org

manintheit.org

How to Create Red Hat HA Cluster Part -III

In this post, we will do the actual work to finish cluster configuration such as starting cluster services, configuration of the clustered LVM disk, creating Fail-over domain, resources, and service groups. Two-Node Cluster As it is indicated in the previous posts two-node cluster is a special form of a cluster because of split-brain situations. So we will add special configuration for two-node cluster.

# If you are configuring a two-node cluster, you can execute the following command to allow a single node to maintain quorum (for example, if one node fails):
[root@node01 ~]# ccs -h host --setcman two_node=1 expected_votes=1

Starting Cluster Services: After configuring quorum, fencing and special two-node cluster configuration we need to start cluster services in order.

#Start cluster services on each node in order below.
[root@node01 ~]# service cman start
[root@node01 ~]# service clvmd start #if CLVM has been used to create clustered volumes
[root@node01 ~]# service gfs2 start #if you are using Red Hat GFS2
[root@node01 ~]# service rgmanager start #if you using high-availability (HA) services (rgmanager).

Stopping Cluster Services Stopping cluster services in order also important.

[root@node01 ~]# service rgmanager stop #if you using high-availability (HA) services (rgmanager).
[root@node01 ~]# service gfs2 stop #if you are using Red Hat GFS2
[root@node01 ~]# umount -at gfs2 #if you are using Red Hat GFS2 in conjunction with rgmanager, to ensure that any GFS2 files mounted during rgmanager startup (but not unmounted during shutdown) were also unmounted.
[root@node01 ~]# service clvmd stop #if CLVM has been used to create clustered volumes
[root@node01 ~]# service cman stop

Test: After starting cluster services, we can check if our nodes are available. To check it we use clustat command. For now, we do not have any fail-over domain, resource, service group.

[root@node01 ~]# clustat 
Cluster Status for ankara-cluster @ Thu Jan 11 20:57:34 2018
Member Status: Quorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 node01-hb.cls.local                         1 Online, Local, rgmanager
 node02-hb.cls.local                         2 Online, rgmanager
 /dev/block/8:16                             0 Online, Quorum Disk

Clustered LVM Configuration

It is very important to configure disk devices  with the correct configuration due to the fact that any node may corrupt the data on the shared disk because of the nodes in the cluster may access disk media at the same time for writing.

To use LVM on the cluster systems do not forget to change locking_type on the /etc/lvm/lvm.conf

Change locking_type from 1 to locking_type=3

[root@node01 ~]# grep locking_type /etc/lvm/lvm.conf 
	# Configuration option global/locking_type.
	#     when to use it instead of the configured locking_type.
	locking_type = 3
	# Attempt to use built-in cluster locking if locking_type 2 fails.
	# Use locking_type 1 (local) if locking_type 2 or 3 fail.
	# locking_type 1 viz. local file-based locking.
	# The external locking library to use for locking_type 2.

Restart clvmd service on each node.

[root@node01 ~]# service clvmd restart

Initializing disks for the clustered LVM

#You can run below command any of the one node
[root@node01 ~]# pvcreate /dev/sdc
[root@node01 ~]# pvcreate /dev/sdd

Creating a Volume group

Creating a volume group for the clustered environment is almost the same as creating volume group for unclustered environment. Only difference we use -cy to specify clustered system.

[root@node01 ~]# vgcreate -cy vgcls_httpd /dev/sdc
[root@node01 ~]# vgcreate -cy vgcls_nfs /dev/sdd

Creating a Logical Volumes and Activating

[root@node01 ~]# lvcreate -n lv_httpd -l 100%vg vgcls_httpd
  Logical volume "lv_httpd" created.
[root@node01 ~]# lvcreate -n lvcls_nfs -l 100%vg vgcls_nfs
  Logical volume "lvcls_nfs" created.
[root@node01 ~]# lvchange -ay vgcls_httpd/lv_httpd
[root@node01 ~]# lvchange -ay vgcls_nfs/lvcls_nfs

#Scanning and rebuilding lvm caches
[root@node01 ~]# vgscan --mknodes -v

Creating GFS2 Files System Even though we have cluster-aware underlying device, we still need a file system on OS level.

[root@node01 ~]# mkfs.gfs2 -t ankara-cluster:httpdgfs2 -j 2 -J 64 /dev/vgcls_httpd/lv_httpd
Device:                    /dev/vgcls_httpd/lv_httpd
Blocksize:                 4096
Device Size                1.00 GB (261120 blocks)
Filesystem Size:           1.00 GB (261118 blocks)
Journals:                  2
Resource Groups:           4
Locking Protocol:          "lock_dlm"
Lock Table:                "ankara-cluster:httpdgfs2"
UUID:                      996a0360-1895-2c53-b4ed-876151027b61

Creating a Failover-Domain: Fail-over domain determines which cluster nodes are allowed to run which services on the cluster.

#Creating a failover domain
[root@node01 ~]# ccs -h localhost --addfailoverdomain name=httpd

Listing Resources We have quite of resource types. You can list available resource options.

[root@node01 ~]# ccs -h localhost --lsserviceopts
service - Defines a service (resource group).
ASEHAagent - Sybase ASE Failover Instance
SAPDatabase - Manages any SAP database (based on Oracle, MaxDB, or DB2)
SAPInstance - SAP instance resource agent
apache - Defines an Apache web server
bind-mount - Defines a bind mount.
clusterfs - Defines a cluster file system mount.
fs - Defines a file system mount.
ip - This is an IP address.
lvm - LVM Failover script
mysql - Defines a MySQL database server
named - Defines an instance of named server
netfs - Defines an NFS/CIFS file system mount.
nfsclient - Defines an NFS client.
nfsexport - This defines an NFS export.
nfsserver - This defines an NFS server resource.
openldap - Defines an Open LDAP server
oracledb - Oracle 10g/11g Failover Instance
oradg - Oracle Data Guard Failover Instance
orainstance - Oracle 10g Failover Instance
oralistener - Oracle 10g Listener Instance
postgres-8 - Defines a PostgreSQL server
samba - Dynamic smbd/nmbd resource agent
script - LSB-compliant init script as a clustered resource.
tomcat-6 - Defines a Tomcat server
vm - Defines a Virtual Machine

Adding Resources

[root@node01 ~]# ccs -h localhost --addresource clusterfs name=httpdgfs2 fstype=gfs2 mountpoint=/data/httpd device=UUID="996a0360-1895-2c53-b4ed-876151027b61"
[root@node01 ~]# ccs -h localhost --addresource ip address=192.168.122.10 monitor_link=yes sleeptime=10

Listing Available Service Options

[root@node01 ~]# ccs -h localhost --lsserviceopts
service - Defines a service (resource group).
ASEHAagent - Sybase ASE Failover Instance
SAPDatabase - Manages any SAP database (based on Oracle, MaxDB, or DB2)
SAPInstance - SAP instance resource agent
apache - Defines an Apache web server
bind-mount - Defines a bind mount.
clusterfs - Defines a cluster file system mount.
fs - Defines a file system mount.
ip - This is an IP address.
lvm - LVM Failover script
mysql - Defines a MySQL database server
named - Defines an instance of named server
netfs - Defines an NFS/CIFS file system mount.
nfsclient - Defines an NFS client.
nfsexport - This defines an NFS export.
nfsserver - This defines an NFS server resource.
openldap - Defines an Open LDAP server
oracledb - Oracle 10g/11g Failover Instance
oradg - Oracle Data Guard Failover Instance
orainstance - Oracle 10g Failover Instance
oralistener - Oracle 10g Listener Instance
postgres-8 - Defines a PostgreSQL server
samba - Dynamic smbd/nmbd resource agent
script - LSB-compliant init script as a clustered resource.
tomcat-6 - Defines a Tomcat server
vm - Defines a Virtual Machine

Creating a Service Group The VIP address will be our parent resource which will depend on filesystem. Therefore, the VIP will only be created if the filesystem successfully mount. Pay careful attention to the order in which the service dependencies are added. Bear in mind that the cluster starts services from the bottom to top. Hence, all leave nodes need to be started before the parent node can start.

#Creating a service group
[root@node01 ~]# ccs -h localhost --addservice httpd-resources domain=httpd recovery=relocate
[root@node01 ~]# ccs -h localhost --addsubservice httpd-resources ip ref=192.168.122.10
[root@node01 ~]# ccs -h localhost --addsubservice httpd-resources ip:clusterfs ref=httpdgfs2

Finally sync cluster configuration to nodes.

[root@node01 ~]# ccs -h localhost --sync --activate

Experiment

[root@node01 ~]# clustat 
Cluster Status for ankara-cluster @ Thu Jan 11 22:08:38 2018
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 node01-hb.cls.local                                                 1 Online, Local, rgmanager
 node02-hb.cls.local                                                 2 Online, rgmanager
 /dev/block/8:16                                                     0 Online, Quorum Disk

 Service Name                                                  Owner (Last)                                                  State         
 ------- ----                                                  ----- ------                                                  -----         
 service:httpd-resources                                       node01-hb.cls.local

Cluster Configuration on /etc/cluster/cluster.conf

<?xml version="1.0"?>
<cluster config_version="127" name="ankara-cluster">
	<fence_daemon/>
	<clusternodes>
		<clusternode name="node01-hb.cls.local" nodeid="1">
			<fence>
				<method name="FMET_XVM">
					<device domain="node01" name="FDEV_XVM1"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="node02-hb.cls.local" nodeid="2">
			<fence>
				<method name="FMET_XVM">
					<device domain="node02" name="FDEV_XVM2"/>
				</method>
			</fence>
		</clusternode>
	</clusternodes>
	<cman expected_votes="1" two_node="1"/>
	<fencedevices>
		<fencedevice agent="fence_xvm" name="FDEV_XVM1"/>
		<fencedevice agent="fence_xvm" name="FDEV_XVM2"/>
	</fencedevices>
	<rm>
		<failoverdomains>
			<failoverdomain name="name=httpd" nofailback="0" ordered="0" restricted="0"/>
		</failoverdomains>
		<resources>
			<clusterfs device="UUID=996a0360-1895-2c53-b4ed-876151027b61" fstype="gfs2" mountpoint="/data/httpd" name="httpdgfs2"/>
			<ip address="192.168.122.10" monitor_link="yes" sleeptime="10"/>
		</resources>
		<service domain="httpd" name="httpd-resources" recovery="relocate">
			<ip ref="192.168.122.10">
				<clusterfs ref="httpdgfs2"/>
			</ip>
		</service>
	</rm>
	<quorumd label="qdisk"/>
</cluster>

File system The disk which is /dev/mapper/vgcls_httpd-lv_httpd automatically mounted  on a /data/httpd without configuring fstab file.

df -hP
/dev/mapper/vgcls_httpd-lv_httpd 1020M 131M 890M 13% /data/httpd

IP As you see, Cluster ip (192.168.122.10) listed on the active node which is node01.

ip a s
2: eth0: <BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdisc pfifo\_fast state UP qlen 1000
    link/ether 52:54:00:2a:26:69 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.100/24 brd 192.168.122.255 scope global eth0
    inet 192.168.122.10/24 scope global secondary eth0

Apache Configuration Only thing we need to do is installing Apache on each node(failover-domains) and putting your web pages (DocumentRoot) into the /data/httpd mount point on the active node. You also need to sync your apache configurations  (/etc/httpd.conf.d)  to the other Apache server related nodes in the cluster whenever you modify your Apache configuration. You can use rsync or scp for this.

Sample Apache Configuration on Each Apache related node.

[root@node01 ~]# cat /etc/httpd/conf.d/00-default.conf

<Directory /data/httpd/default/www>
	#Require all granted
	Allow from all
</Directory>

<VirtualHost _default_:80>
	ServerName ankara-cluster
	DocumentRoot /data/httpd/default/www
</virtualHost>

/etc/hosts Contents of the /etc/hosts file on each cluster node and user Workstations who need to access web pages and nfs export.

192.168.122.100 node01
192.168.123.100 node01-hb
192.168.122.200 node02
192.168.123.200 node02-hb

192.168.122.10 ankara-cluster
192.168.122.20 nfscls

Testing Web Page

[root@oc7133075241 conf.d]# curl http://ankara-cluster
<h1> Hello Ankara-Cluster</h1>
<h2>Tue, 14 2017</h2>

That is all for now. Next post we will configure HA NFS on the cluster environment.

REFERENCES

https://bigthinkingapplied.com/creating-a-ha-cluster-with-red-hat-cluster-suite/ https://www.redhat.com/en https://wiki.clusterlabs.org/wiki/Guest_Fencing

Posted by:

admin

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.