VLAN Creation on KVM-I

Creating a VLAN on KVM requires more raw networking knowledge in comparison to VMware world. KVM requires some Linux networking knowledge beside general understanding of computer networks. I did¬† not see more information on the Internet about that. To fill this gap I write this post. ūüôā Actually there are more than one methods creating a VLAN on KVM. In this post I will show the first method.¬† In this method¬† sub-interfaces are created in the bridge NOT in the physical NIC interface. Doing so, Vlan tags¬† wont be stripped-off or Vlan tags wont be embedded in the physical interface but bridge. I use Ubuntu 16.04 for the KVM host. In this post, I will use KVM and libvirt interchangeable.

root@ankara:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial

Loading 8021q module

First thing we need to load 8021q module on the KVM host in order to encapsulate or d-encapsulate IEEE 802.1Q type Vlan.

root@ankara:~# modprobe 8021q

To load module automatically on boot. Create a file 8021q.conf in the /etc/modules-load.d/ and add 8021q

root@ankara:~# cat /etc/modules-load.d/8021q.conf 
8021q

Creating a Bridge(s)

In order to create a vlan(trunk and access ) we need to crate bridge(s) and tell the system tag the frames. In Linux, to create tagged frame we use vconfig command. We need to install vlan package to use it.

root@ankara:~# apt-get install vlan

Note: Creating a bridge with this method is NOT persistent. To make it persistent you need do add configuration to /etc/network/interfaces file. Because of plenty of tutorials about that, I do not explain it here.

root@ankara:~# brctl addbr br0
root@ankara:~# vconfig add br0 30 #subinterface(vlan30)
root@ankara:~# vconfig add br0 40 #subinterface(vlan40)
root@ankara:~# brctl addbr vlan30
root@ankara:~# brctl addbr vlan40
root@ankara:~# brctl addif vlan30 br0.30 #vlan30
root@ankara:~# brctl addif vlan40 br0.40 #vlan40

You can see network interfaces after creating bridges and sub-interfaces on KVM Host.

root@ankara:~# ip link show 
...(omitted some output)
11: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
12: br0.30@br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master vlan30 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
13: br0.40@br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master vlan40 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
14: vlan30: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
15: vlan40: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff

We also need to up links.

root@ankara:/etc/libvirt/qemu/networks# ip link set br0 up
root@ankara:/etc/libvirt/qemu/networks# ip link set br0.30 up
root@ankara:/etc/libvirt/qemu/networks# ip link set br0.40 up
root@ankara:/etc/libvirt/qemu/networks# ip link set vlan30 up
root@ankara:/etc/libvirt/qemu/networks# ip link set vlan40 up

Final bridge status

root@ankara:~# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.000000000000	no			
vlan30		8000.000000000000	no		br0.30
vlan40		8000.000000000000	no		br0.40

Defining Bridges on KVM.

After creating bridges, We also need to define bridges to our hypervisor to use it. I will create three configuration files for the br0, vlan30 and vlan40 successively in the /etc/libvirt/qemu/networks folder.

br0.xml

<network>
  <name>br0</name>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

vlan30.xml

<network>
  <name>vlan30</name>
  <forward mode='bridge'/>
  <bridge name='vlan30'/>
</network>

vlan40.xml

<network>
  <name>vlan40</name>
  <forward mode='bridge'/>
  <bridge name='vlan40'/>
</network>

 

root@ankara:/etc/libvirt/qemu/networks# virsh net-define br0.xml
root@ankara:/etc/libvirt/qemu/networks# virsh net-define vlan30.xml
root@ankara:/etc/libvirt/qemu/networks# virsh net-define vlan40.xml
root@ankara:/etc/libvirt/qemu/networks# virsh net-start br0
root@ankara:/etc/libvirt/qemu/networks# virsh net-start vlan30
root@ankara:/etc/libvirt/qemu/networks# virsh net-start vlan40
#to auto start on boot.
root@ankara:/etc/libvirt/qemu/networks# virsh net-autostart br0
root@ankara:/etc/libvirt/qemu/networks# virsh net-autostart vlan30
root@ankara:/etc/libvirt/qemu/networks# virsh net-autostart vlan40

Checking bridges

root@ankara:/etc/libvirt/qemu/networks# virsh  net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 br0                  active     yes           yes
 vlan30               active     yes           yes
 vlan40               active     yes           yes

So far so good ?

You may confuse due to the fact that we did many things so far. I hope below figure gives you a better understanding what we did so far. It is depicted below figure how our network looks like. Only think that I did not do is adding physical interface to bridge br0. In this post, KVM guests will not connect to the Internet. According to this design we do not need to setup any VLAN configuration on¬† the KVM virtual guests. It has all handled by br0.30 and br0.40–Any outgoing packet from VLAN30 network will be tagged by the br0.30 sub-interface. Any incoming tagged packet to VLAN30 network will be stripped-off by the br0.30 sub-interface. It is the same as VLAN40 network.

 

 

 

 

 

 

 

 

 

 

 

Experiments

I captured the packages on br0.30 interface and br0 bridges to check, if  vlans works as expected.

Output br0.30(we see incoming tagged icmp request stripped-off by the br0.30 we see untagged frames)

root@ankara:/etc/libvirt/qemu/networks# tcpdump -i br0.30 -e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0.30, link-type EN10MB (Ethernet), capture size 262144 bytes
15:06:46.125189 70:54:d2:99:56:c0 (oui Unknown) > 52:54:00:43:40:b7 (oui Unknown), ethertype IPv4 (0x0800), length 98: 172.16.30.50 > 172.16.30.10: ICMP echo request, id 2453, seq 520, length 64
15:06:46.125429 52:54:00:43:40:b7 (oui Unknown) > 70:54:d2:99:56:c0 (oui Unknown), ethertype IPv4 (0x0800), length 98: 172.16.30.10 > 172.16.30.50: ICMP echo reply, id 2453, seq 520, length 64
15:06:47.149216 70:54:d2:99:56:c0 (oui Unknown) > 52:54:00:43:40:b7 (oui Unknown), ethertype IPv4 (0x0800), length 98: 172.16.30.50 > 172.16.30.10: ICMP echo request, id 2453, seq 521, length 64
15:06:47.149530 52:54:00:43:40:b7 (oui Unknown) > 70:54:d2:99:56:c0 (oui Unknown), ethertype IPv4 (0x0800), length 98: 172.16.30.10 > 172.16.30.50: ICMP echo reply, id 2453, seq 521, length 64

 

Output on the br0 (we see tagged 802.1q encapsulation vlan30)

root@ankara:/etc/libvirt/qemu/networks# tcpdump -i br0 -e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:06:58.413319 70:54:d2:99:56:c0 (oui Unknown) > 52:54:00:43:40:b7 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 30, p 0, ethertype IPv4, 172.16.30.50 > 172.16.30.10: ICMP echo request, id 2453, seq 532, length 64
15:06:58.413564 52:54:00:43:40:b7 (oui Unknown) > 70:54:d2:99:56:c0 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 30....

 

According to the figure above, hosts on the VLAN30 and hosts on the VLAN40 can not communicate each other as we do not have L3 device for Inter Vlan Routing. Next post, I will provision virtual L3 device which will be VyOS(Vyatta) on the KVM. I will add two network interfaces on it. –Connect one interface to the br0(trunk port) and¬† the other interface¬† to the physical NIC for the Internet connection.

Configuring PAM for crontab

Because of some strict rules such as PCI DSS compliances, we could not set password for root user to “never expire”. But, when password expires for any account in the Linux systems crontab jobs will not work. Unless, you configure the PAM to run it, or extending root password. Or you need to buy some proprietary software to change root password periodically in the defined period of time.

By the way, even your root password has expired, switching to root user from non-root user will still work.(If  NOPASSWD option added in sudoers).

In this post, PAM configuration will be changed to elevate to run crontab jobs, even root password has expired.

Problem:

/var/log/crond

Nov 11 20:45:01 node01 crond[6057]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)

/var/log/secure

Nov 12 11:04:44 node01 crontab: pam_unix(crond:account): expired password for user root (root enforced)
Nov 12 11:05:01 node01 crond[8459]: pam_unix(crond:account): expired password for user root (root enforced)

crontab -l output

[root@node01 log]# crontab -l

Authentication token is no longer valid; new one required
You (root) are not allowed to access to (crontab) because of pam configuration.

Solutions:

There are two ways to solve this issue. The first one is the extend root password, or set it to “never expire”. After that your crontab jobs will work again. The second one, you can configure PAM in /etc/pam.d/crond. In this post we chose second option which is PAM.

You can see contents of the crond configuration file in the /etc/pam.d

#
# The PAM configuration file for the cron daemon
#
#
# No PAM authentication called, auth modules not needed
account    required   pam_access.so
account    include    password-auth
session    required   pam_loginuid.so
session    include    password-auth
auth       include    password-auth

Only thing you need to change is the “required” to “sufficient”¬† for the pam_access.so module.

account sufficient pam_access.so

Final configuration

#
# The PAM configuration file for the cron daemon
#
#
# No PAM authentication called, auth modules not needed
account    sufficient   pam_access.so
account    include    password-auth
session    required   pam_loginuid.so
session    include    password-auth
auth       include    password-auth

Right after change has been applied, crontab should work.

[root@node01 log]# crontab -l
*	*	*	*	*	/root/re.sh

 

Nov 12 11:19:01 node01 CROND[8508]: (root) CMD (/root/re.sh)
Nov 12 11:20:01 node01 CROND[8510]: (root) CMD (/root/re.sh)

Note: This configuration applied for all users in the system, which means all users whose password expired will also be able to run crontab. If you only want to do it for root user you should add an argument to  module by specifying userid.

According to below configuration, crontab jobs will run only for a root user.

#
# The PAM configuration file for the cron daemon
#
#
# No PAM authentication called, auth modules not needed
account    sufficient pam_succeed_if.so uid = 0 quiet
account    include    password-auth
session    required   pam_loginuid.so
session    include    password-auth
auth       include    password-auth

 

CPU Affinity

What is CPU affinity ?

Binding or unbinding  one or more processes to one or more processors called CPU affinity. It enables that process(es) runs only designated processor(s) rather than any processor. There are two types of CPU affinity, which are soft affinity and hard affinity.

Soft affinity is a mechanism that keeping  process on the same CPU as long as possible otherwise, process is migrated to other processor.

Hard affinity is a mechanism that enables processes can run only on a fixed set of  one or more processor(s).

Benefits of CPU affinity

1- Optimizes cache performance as the process runs on the same CPU core and no cache invalidation happens. (NUMA architecture.)

2- Time-sensitive, and real time  applications. Due to the fact that those applications will run only designated cores. And scheduler runs other applications on the other processors.

How to Implement CPU affinity in Linux ?

In Linux, you can set CPU affinity with taskset or numactl command. In this post I will use taskset command.

taskset -c <cpu core(s)> <program>

To run a nvpy, which is python based note taking application, on the only core 0.

taskset -c 0 nvpy

To set cpu affinity for a running process.

First we need to find pid.

gokay@ankara:~$ pgrep nvpy
26119

Set CPU affinity to core 3 for  running process whose pid is 26119.(nvpy)

gokay@ankara:~$ taskset -pc 3 26119
pid 26119's current affinity list: 0
pid 26119's new affinity list: 3
gokay@ankara:~$

Showing process (26119) and its threads of CPUs currently running on.

gokay@ankara:~$ ps -T -p 26119 -o pid,spid,psr
  PID  SPID PSR
26119 26119   3
26119 26121   0
26119 26122   0
26119 26123   0

 

Note: If you do not know what to do please leave it to OS scheduler. Otherwise your application may suffer from compute resources.