Highly available Load-balancer for Kubernetes Cluster On-Premise – I

In this post, we are going to build highly available HAProxy Load-balancer for our Kubernetes cluster on-premise. For this, HaProxy will be used for external Load-balancer which takes the requests from outside world sends them to Kubernetes worker nodes on which nginx ingress controller listens incoming requests on port 80 and 443.

Another curial software component is Keepalived which provides a highly available HAProxy load-balancer, in case of any of HAProxy loadbalancer is down.

Keepalived is a Robust Virtual Router Redundancy Protocal (VRRP )implementation in GNU/Linux

To build cluster, Ubuntu 18.04.4 used. You can see below diagram how environment looks like.

Installing necessary Software Suits

# sudo apt-get install haproxy
# sudo apt-get install keepalived
# sudo systemctl enable haproxy
# sudo systemctl enable keepalived

Configuring Necessary Kernel Parameters

The below configuration is very important to implement it on both nodes.

In order for the Keepalived service to forward network packets properly to the real servers, each node must have IP forwarding turned on in the kernel. Log in as root and change the line which reads net.ipv4.ip_forward = 0 in /etc/sysctl.conf to the following:

net.ipv4.ip_forward = 1

The changes take effect when you reboot the system. Load balancing in HAProxy and Keepalived at the same time also requires the ability to bind to an IP address that are nonlocal, meaning that it is not assigned to a device on the local system. This allows a running load balancer instance to bind to an IP that is not local for failover. To enable, edit the line in /etc/sysctl.conf that reads net.ipv4.ip_nonlocal_bind to the following:

net.ipv4.ip_nonlocal_bind = 1

The changes take effect when you reboot the system.

Configuring Keepalived on both nodes

Some keepalived settings has to be changed accordingly in the second node. So, check the commented lines in the keepalived.conf

global_defs {
   notification_email {
     admin@manintheit.org
   }
   notification_email_from keepalived@manintheit.org
   smtp_server localhost
   smtp_connect_timeout 30
   router_id ha1 #router_id ha2 on the second node(ha2)
   vrrp_skip_check_adv_addr
   vrrp_garp_interval 0.5
   vrrp_garp_master_delay 1
   vrrp_garp_master_repeat 5
   vrrp_gna_interval 0
   enable_script_security
   script_user root
   vrrp_no_swap
   checker_no_swap
}

# Script used to check if HAProxy is running
vrrp_script check_haproxy {
       script "/usr/bin/pgrep haproxy 2>&1 >/dev/null"
        interval 1
        fall 2
        rise 2
}

# Virtual interface
vrrp_instance VI_01 {
        state MASTER #state BACKUP on the second node(ha2)
        interface enp1s0
        virtual_router_id 120
        priority 101  #priority 100 on the second node(ha2). Higher number wins.
        nopreempt
        advert_int 1
        unicast_src_ip 10.5.100.51  #unicast_src_ip 10.5.100.52 on the second node(ha2)
        unicast_peer {
                10.5.100.52    #unicast_peer 10.5.100.51 on the second (ha2)
        }
        virtual_ipaddress {
                10.5.100.50/24 dev enp1s0 label enp1s0:ha-vip1
        }
        authentication {
                auth_type PASS
                auth_pass MANINTHEIT
        }
        track_script {
                check_haproxy
        }
}

HAProxy Config

#Default configurations in the haproxy.cfg has been omitted.

frontend stats
    bind 10.5.100.51:9000
    mode http
    maxconn 10
    stats enable
    stats show-node
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /hastats
    stats auth haadmin:haadmin

frontend k8s-service-pool
  mode tcp
  bind 10.5.100.50:80
  default_backend k8s-service-backend

backend k8s-service-backend
    mode tcp
    balance source
    server k8s-worker-01 10.5.100.21 check port 80 inter 10s rise 1 fall 2 
    server k8s-worker-02 10.5.100.22 check port 80 inter 10s rise 1 fall 2
    server k8s-worker-03 10.5.100.23 check port 80 inter 10s rise 1 fall 2

Restart the service keepalived and haproxy on both nodes.

# sudo systemctl restart haproxy
# sudo systemctl restart keepalived

Experiment:

1- Lets check with tcpdump utility if master node sends VRRP advertisement packets to every second to all members of VRRP group.

# tcpdump proto 112 -n

2- Lets check interface IPs; As you see, first node(ha1) looks active node as it register to VIP 10.5.100.50.

Second Part of the Post will be published soon, Stay healthy!

Hourly logrotation

Sometimes, you need to rotate your logs hourly instead of daily or weekly, if you have a big virtual environment many things needs to be logged. Sometimes daily logs are so huge that you need hourly log-rotation. For this one you need to customize some of the settings in your central syslog server. You can find the sample steps below to create logrotate configuration that rotates the logs hourly.

Steps:

1 – Copy /etc/cron.daily/logrotate to /etc/cron.hourly and set it as executable.

# cp /etc/cron.daily/logrotate /etc/cron.hourly
# chmod u+x /etc/cron.hourly/logrotate

2- Create a folder logrotate.hourly.conf in /etc

# mkdir -p /etc/logrotate.hourly.conf

3- Modify the file logrotate in the /etc/cron.hourly based on your needs. See below for sample.

#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.hourly.conf/example
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit $EXITVALUE

3- Create your logrotate script in the folder /etc/logrotate.hourly.conf

For this post, we named it as ‘example’. (We also specified it in the configuration /etc/logrotate.hourly.conf/logrotate)

/opt/logs/[2-9][0-9][0-9][0-9]/[0-9][0-9]/[0-9][0-9]/*.log {
    notifempty
    compress
    maxage 60
    rotate 200
    create 0600 root root
    size 4G
    postrotate
        /usr/bin/systemctl reload syslog-ng > /dev/null
    endscript
}

  • You may need to tune up rotate, size and maxage options base on your needs.

LACP Configuration with Cumulux VX Virtual Appliances

In this post LACP will be configured on Cumulus VX virtual appliance. Test simulated on GNS3 Network simulation.

Sample Topology

This is the sample network topology to test LACP.

Configuration on Both Virtual Appliances for LACP

In this configuration, swp1 and swp2 ports used as a slave ports of LAGG0 on both switches.

cumulus@cumulus:~$ net add bond LAGG0 bond slaves swp1,2
cumulus@cumulus:~$ net add bond LAGG0 bond mode 802.3ad
cumulus@cumulus:~$ net add bond LAGG0 bond lacp-rate slow
cumulus@cumulus:~$ net add bond LAGG0 bond miimon 100

Configuration on Both Virtual Appliances for VLAN configuration

In below configuration, VLAN 10 added to bridge(on cumulus vlan-aware bridge) and swp3 configured as an access port with VLAN id 10.

cumulus@cumulus:~$ net add bridge bridge vlan-protocol 802.1ad
cumulus@cumulus:~$ net add bridge bridge ports LAGG0
cumulus@cumulus:~$ net add bridge bridge ports swp3
cumulus@cumulus:~$ net add bridge bridge vids 10
cumulus@cumulus:~$ net add interface swp3 bridge access 10

Experiment

It is also tested that after one of the link has been cut, host still able to ping to other end without any package drop.