Kubernetes MetalLB Loadbalancer with BGP mode
In this post, MetalLB will be used on-premise as a Load Balancer with BGP to expose services on Kubernetes to outsideworld. As you know, if you have applications run on Kubernetes which is on your on-premise, that needs to be exposed to the outsideworld, should use LoadBalancer which is a bit tricky. it is quite easy if your applications in cloud environment such as AWS, Azure, or Google Cloud. Otherwise, you have to do similar in your on-premise network, with MetalLB or HAproxy. You can see my previous post how to achieve this with HAProxy here . For this post, we are going to achieve the same thing with MetalLB. You can see the Figure below, basic concept of MetalLB.
MetalLB in BGP mode.
I am assuming that you have Kubernetes installed on your Infrastructure. You can start configuring Metal-lb on your Kubernetes with the instruction in the link . If you follow the steps in the link you should get the results like below. MetalLB runs speaker pods in each worker and master nodes and controller pod in one of the node. As I understood in the metalLB website each speaker node establishes a BGP connection with router.
[[email protected] ~]$ kubectl get pods -n metallb-system
NAME READY STATUS RESTARTS AGE
controller-57f648cb96-ncbww 1/1 Running 1 3h8m
speaker-5n2c4 1/1 Running 0 3h8m
speaker-7mwx2 1/1 Running 8 3h8m
speaker-c7fgm 1/1 Running 0 3h8m
speaker-vxcbd 1/1 Running 0 3h8m
Next step is creating a configmap named ‘config’ in the namespace metallb-system.
Creating a Configmap named config in the namespace metallb-system
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 10.5.100.254
peer-asn: 65000
my-asn: 65000
address-pools:
- name: default
protocol: bgp
avoid-buggy-ips: true
addresses:
- 10.5.120.0/24
I faced a problem that, without adding an option avoid-buggy-ips: true, MetalLB allocated the ip 10.5.120.0, which is network address.
Configuring BGP on “Intelligent” Home router
I am using the physical router EdgeRouterX in my home lab, which is able to run BGP protocol with ECMP.
configure
set protocols bgp 65000 parameters router-id 10.5.100.254
set protocols bgp 65000 neighbor 10.5.100.20 remote-as 65000
set protocols bgp 65000 neighbor 10.5.100.21 remote-as 65000
set protocols bgp 65000 neighbor 10.5.100.22 remote-as 65000
set protocols bgp 65000 maximum-paths ibgp 6
commit
save
exit
As it is my home lab, I just picked one of the AS number(65000) that is in the private range. For more info on ASN number, check here .
Experiment:
After deploying MetalLB and configuring the Router with BGP, we can do some checks, if BGP peering successfully between the Kubernetes worker nodes and my Home router.
[email protected]:~$ show ip bgp summary
BGP router identifier 10.5.100.254, local AS number 65000
BGP table version is 27
1 BGP AS-PATH entries
0 BGP community entries
1 Configured ebgp ECMP multipath: Currently set at 1
6 Configured ibgp ECMP multipath: Currently set at 6
Neighbor V AS MsgRcv MsgSen TblVer InQ OutQ Up/Down State/PfxRcd
10.5.100.20 4 65000 494 485 27 0 0 03:25:32 1
10.5.100.21 4 65000 409 423 27 0 0 03:17:29 1
10.5.100.22 4 65000 494 486 27 0 0 03:25:32 1
Total number of neighbors 3
Total number of Established sessions 3
Checking the logs in one of speaker pod.
[[email protected] ~]$ kubectl logs -f -n metallb-system speaker-5n2c4
{"caller":"bgp_controller.go:232","event":"updatedAdvertisements","ip":"10.5.120.1","msg":"making advertisements using BGP","numAds":1,"pool":"default","protocol":"bgp","service":"default/nginx","ts":"2020-08-30T09:59:48.873159314Z"}
Creating a Sample Deployment
Last step is to creating a sample deployment to check if MetalLB works as expected.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1
ports:
- name: http
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer
[[email protected] ~]$ kubectl get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 1/1 1 1 145m
[[email protected] ~]$ kubectl get svc nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx LoadBalancer 10.96.136.46 10.5.120.1 8080:30169/TCP 88m
[email protected]:~$ curl http://10.5.120.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
The last but not least, I had a problem while trying to access to application from KVM host. VMs are able to reach without an issue. I solved the problem, once I added default gateway for my ‘Intelligent’ router.
default via 10.5.100.254 dev br100*** > My 'Intelligent router'
default via 192.168.0.1 dev wlp4s0 proto dhcp metric 600*** > 'My Wifi Router'