SSL Client Certificate Authentication with Apache

Creating CA Certificate

We use this certificate for only signing certificates that we use for the clients and our web servers. It should be kept very secure. If it is disclosed other certificates signed with this certificate will be disclosed as well.

openssl genrsa -des3 -out ca.key 4096
openssl req -new -x509 -days 365 -key ca.key -out ca.crt

Creating a Key and CSR for the Client

Creating a client certificate is the same as creating Server certificate.

openssl req -newkey rsa:2048 -nodes -keyout client.key -out client.csr

Signing the client certificate with previously created CA.

Not: Do not forget to change serial each time you sign new certificate, otherwise may get serial conflict error in the web browsers.

[root@centos7 certs]# openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt
Signature ok
subject=/C=TR/L=Default City/O=Client Certificate/CN=Client Certificate
Getting CA Private Key
Enter pass phrase for ca.key:

Creating a Key and CSR for the Server(Apache Virtual Host

openssl req -newkey rsa:2048 -nodes -keyout ankara.key -out ankara.csr

Signing Server Certificate with previously created CA.

Do not forget to change serial number. As it may conflict with existing one.

openssl x509 -req -days 365 -in ankara.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out ankara.crt

Apache Configuration for the Authentication with Client Certificate

This sample configuration shows how to force server to request client certificate.

<Directory /srv/ankara/www>
	Require all granted

<VirtualHost *:443>
	SSLEngine On
	SSLCertificateFile /etc/httpd/conf.d/certs/ankara.crt
	SSLCertificateKeyFile /etc/httpd/conf.d/certs/ankara.key
	DocumentRoot /srv/ankara/www 
	SSLVerifyClient require
	SSLVerifyDepth 5
	SSLCACertificateFile "/etc/httpd/conf.d/certs/ca.crt"

The depth actually is the maximum number of intermediate certificate issuers, i.e. the number of CA certificates which are max allowed to be followed while verifying the client certificate. A depth of 0 means that self-signed client certificates are accepted only, the default depth of 1 means the client certificate can be self-signed or has to be signed by a CA which is directly known to the server (i.e. the CA’s certificate is under SSLCACertificatePath), etc.


Experimenting with Curl

Without specifying the client certificate

gokay@ankara:~/certs$ curl -v
* Rebuilt URL to:
* Trying
* Connected to ( port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: Handshake failed
* Closing connection 0
curl: (35) gnutls_handshake() failed: Handshake failed


With client certificate

gokay@ankara:~/certs$ curl --key client.key --cert client.crt --cacert ca.crt -v
* Rebuilt URL to:
* Trying
* Connected to ( port 443 (#0)
* found 1 certificates in ca.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #1
* subject: C=TR,L=Default City,O=Ankara LTD,
* start date: Sun, 24 Dec 2017 10:00:20 GMT
* expire date: Mon, 24 Dec 2018 10:00:20 GMT
* issuer: C=TR,L=Default City,O=BlueTech CA,OU=CA,CN=BlueTech CA
* compression: NULL
* ALPN, server did not agree to a protocol
> GET / HTTP/1.1
> Host:
> User-Agent: curl/7.47.0
> Accept: */*
< HTTP/1.1 200 OK
< Date: Sun, 24 Dec 2017 10:21:15 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< Last-Modified: Sun, 24 Dec 2017 10:19:51 GMT
< ETag: "3b-5611363e4a8e0"
< Accept-Ranges: bytes
< Content-Length: 59
< Content-Type: text/html; charset=UTF-8
<h1>My Secure Page Ankara</h1>
* Connection #0 to host left intact


Converting certificate and to pkcs12 format

If you want to import a certificate to a web browser, you have to convert your existing certificate other than PEM format. For the Mozilla Firefox, you need to convert it  to pkcs12 format.

openssl pkcs12 -export -out ankara.pfx -inkey ankara.key -in ankara.crt -certfile ca.crt

Only thing we need to do it import ankara.pfx to our browser.

Disabling Client Certificate Authentication

Comment out last three lines between the <VirtualHost> and </VirtualHost> directive.

<Directory /srv/ankara/www>
	Require all granted

<VirtualHost *:443>
	SSLEngine On
	SSLCertificateFile /etc/httpd/conf.d/certs/ankara.crt
	SSLCertificateKeyFile /etc/httpd/conf.d/certs/ankara.key
	DocumentRoot /srv/ankara/www 
	#SSLVerifyClient require
	#SSLVerifyDepth 5
	#SSLCACertificateFile "/etc/httpd/conf.d/certs/ca.crt"


Only thing that we need to do is specifying CA certificate or providing  -k option to curl for insecure SSL connection.

gokay@ankara:~/certs$ curl  --cacert ca.crt 
<h1>My Secure Page Ankara</h1>



Fake Kernel Panic for Testing(Magic SysRq)

Sometimes we need to crash the system for testing of cluster  fencing functionality  and fail-over functionality.  Actually in Linux we can create a fake kernel panic by issuing below commands.


root@ankara # echo 1 > /proc/sys/kernel/sysrq  #1 - enable all functions of sysrq
root@ankara # echo "c" > /proc/sysrq-trigger   #


c Will perform a system crash by a NULL pointer dereference. A crash dump will be taken if configured.


System Restore with Rsync

Rsync is one of the  super-duper *nix utility for synchronizing or transferring files among the computers. It has so much functionality that I could not explain it to you in one post. In this post, I will use rsync utility to backup and restore my system. In this activity, I am using CentOS7(virtual guest) on KVM host.

Backing up Whole System

Before destroy the running system, I will backup whole running system to my remote backup server with rsync. I am excluding some of the folders as they are populated at boot time. By using below rsync options, I will able to backup the system with extended attributes and SELinux.

backup # rsync -vaHAXSz --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} root@centos7:/ /opt/backup/centos7/


-H, --hard-links      preserve hard links
-A, --acls            preserve ACLs (implies -p)
-X, --xattrs          preserve extended attributes
-S, --sparse          handle sparse files efficiently
-v, --verbose         increase verbosity
-z, --compress        compress file data during the transfer
-a, --archive         archive mode; equals -rlptgoD (no -H,-A,-X)

The system has backed-up,  while it was running. We are safe now.

Restoring the Corrupted System.

To restore corrupted system from the existing backup we need couple of things to do before the actual operations.

1- Attach live CentOS7 ISO image and the boot the system from it.

2- After your system booted, configure your network and routing settings temporarily to connect to the Backup Server.

ifconfig <eth-name> <ip addresss> netmask <netmask> up

ip route add default via <gateway> dev <eth-name>

I configure my system as below.

live#ifconfig ens3 netmask up
live#ip route add default via dev ens3

3- Ping your backup server if it is accessible from the live system.

4- Partition the disk with fdisk utility. Or gdisk utility, if you want to partition the disk with GPT.

I – For separate boot partition 500MiB is enough. (/dev/sda1)

II- For the root (/) partition.

5- Open-Up the terminal.

As I used lvm disk on my system, I recreate the VGs and LVs from the existing configuration. In order for that, first copy your lvm folder from your backup  server to your live system’s /root partition.

live# rsync -avz  backup@ /root/lvm

I will initialize my physical disk to use lvm and I will use the same uuid as before. To find that uuid I will look into the configuration in the /root/lvm/backup/<vgname> file.

Initializing the disk for LVM with the same uuid.

live# pvcreate --uuid "fpK70u-ZvuO-bgBM-hO67-3M2l-VFFo-0rCWsa" --restorefile /root/lvm/backup/vg_node01 /dev/sda2
Couldn't find device with uuid fpK70u-ZvuO-bgBM-hO67-3M2l-VFFo-0rCWsa.
Physical volume "/dev/sda2" successfully created.

Restoring the VG

live# vgcfgrestore -f /root/lvm/backup/vg_node01 vg_node01
  Restored volume group vg_node01


live# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_root vg_node01 -wi-a----- 6.71g
lv_swap vg_node01 -wi-a----- 816.00m

Building file system ext4 and swap

live# mkfs.ext4 /dev/sda1
live# mkfs.ext4 /dev/mapper/vg_node01-lv_root 
live# mkswap /dev/mapper/vg_node01-lv_swap

Create a folder  for  chroot

live# mkdir /mnt/sysimage

Restoring files from the backup server. (excluding /boot contents)

live# rsync -vaHAXSz --exclude={"/boot/*"}  backup@ /mnt/sysimage/

Mounting pseudo files with –bind option.

live# mount --bind /dev /mnt/sysimage/dev 
live# mount --bind /sys /mnt/sysimage/sys
live# mount --bind /proc /mnt/sysimage/proc


live# chroot /mnt/sysimage /bin/bash

After chroot command applied, all operations that we do will  affect files in the /mnt/sysimage which is our actual system files.

Mounting /boot partition in the chroot

centos7# mount /dev/sda1 /boot

After mounting the /boot partition copy the boot contents from the backup server.

centos7# rsync -vaHAXSz root@ /boot/

Install the MBR

We need to embed stage1 in the first sector(512Byte) of the harddisk.

[root@centos7 boot]# grub2-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.
[root@centos7 boot]# 

Update GRUB2

[root@centos7 boot]# grub2-mkconfig -o /boot/grub2/grub.cfg

Do not forget to change fstab entries

If you are using UUID option  in  the fstab, do not forget to change it with the new UUID. You can find the UUID of the each partition with blkid command.

Umount the partitions before reboot.

Exit from chroot and umount pseudo file system partitions and reboot the host.

centos7# exit
live# umount /mnt/sysimage/dev 
live# umount /mnt/sysimage/sys 
live# umount /mnt/sysimage/proc
live# umount /mnt/sysimage
live# reboot