Finding out per process Swap usage

In this post, I will share with you useful shell script to find per process swap usage on your system.

Method- 1

total_swap=0
for pid in `ls /proc | grep "[[:digit:]]"| sort -n`
do
    if [[ "$pid" -gt "0" && -f "/proc/$pid/status" ]]; then
        pname=`cat /proc/$pid/status|awk -F"Name:" '{print $2}'`
        swap_usage=`cat /proc/$pid/smaps | awk 'BEGIN{total=0}/^Swap:/{total+=$2}END{print total}'`
        echo "$pname($pid): $swap_usage KB"
        total_swap=$((total_swap + swap_usage))
    fi
done
echo "Total Swap: $total_swap Kb"

Method-2

cat /proc/{pid}/status|awk '/VmSwap:/{print $2 " " $3}'

Finding Out Memory Usage without adding shared memory (non-shared):

smem utility.

Unshared memory is reported as the USS (Unique Set Size).

The unshared memory (USS) plus a process’s proportion of shared memory is reported as the PSS (Proportional Set Size).”

The USS and PSS only include physical memory usage. They do not include memory that has been swapped out to disk.”

#add epel-release repo.
[root@rhce ~]# yum install smem
[root@rhce ~]#  smem -u ansible
User     Count     Swap      USS      PSS      RSS
nobody       1      184        0        2       92
colord       1     1544        4        5      192
libstoragemgmt     1      116       40       40       60
rtkit        1      184       48       48      140
avahi        2      448      152      170      448
postfix      2     1900      168      171      420
haproxy      2     2732      264      305      560
dbus         1      432     1980     2010     2364
polkitd      1    10904     2780     2897     3856
root        53   146564    97156   101862   119260
ansible     64   413824   477768   481342   501392

Finding out memory statistics as percentage.

[root@rhce ~]# smem -w -p
Area                           Used      Cache   Noncache
firmware/hardware             0.00%      0.00%      0.00%
kernel image                  0.00%      0.00%      0.00%
kernel dynamic memory        30.13%     10.81%     19.31%
userspace memory             58.54%      3.20%     55.33%
free memory                  11.34%     11.34%      0.00%

Show totals and percentage:

 PID User     Command                         Swap      USS      PSS      RSS
77794 ansible  /usr/libexec/gnome-terminal    0.84%    0.23%    0.23%    0.26%
72241 root     /usr/sbin/rsyslogd -n          1.13%    0.26%    0.28%    0.33%
  678 polkitd  /usr/lib/polkit-1/polkitd -    0.69%    0.29%    0.30%    0.40%
 5688 ansible  /usr/libexec/gnome-settings    0.42%    0.32%    0.33%    0.44%
    1 root     /usr/lib/systemd/systemd --    0.07%    0.36%    0.37%    0.41%
 5919 ansible  /usr/libexec/tracker-store     0.50%    0.45%    0.45%    0.47%
 5827 ansible  /usr/libexec/caribou           0.16%    0.65%    0.65%    0.68%
78786 ansible  evince                         0.00%    0.78%    0.80%    0.85%
78818 ansible  eog                            0.00%    0.83%    0.83%    0.85%
 5900 ansible  nautilus --no-default-windo    0.27%    0.89%    0.92%    0.98%
81293 root     python /bin/smem -t -p         0.00%    1.03%    1.07%    1.16%
78686 ansible  /usr/lib64/libreoffice/prog    4.10%    3.40%    3.41%    3.45%
78843 ansible  /usr/lib/jvm/java-1.8.0-ope    0.00%    4.97%    5.03%    5.12%
 1452 root     /usr/bin/Xorg :0 -backgroun    2.21%    5.93%    6.13%    6.34%
78519 ansible  /usr/lib64/firefox/firefox     2.02%   10.02%   10.03%   10.12%
 5733 ansible  /usr/bin/gnome-shell           8.21%   24.20%   24.25%   24.41%
-------------------------------------------------------------------------------
  130 11                                     36.78%   58.20%   59.07%   63.11%

Show as  a Chart:

To show as  a chart you have to install matpilotlib package

[root@rhce ~]# yum install python-matplotlib.x86_64

Plotting specific process: 

smem --userfilter="haproxy" --bar name -c"pss rss"
[root@rhce ~]# pidof haproxy
76708 76706

 

 

 

 

 

 

 

 

 

 

 

Plotting Swap usage:

smem  --processfilter="gnome-shell" --bar name -c "pss uss swap"

 

 

 

 

 

 

 

 

 

 

 

L7 LB with HAProxy

HaProxy is a free and reliable software solution for high availability and load-balancing for TCP and HTTP applications. HaProxy has been used many Fortune companies for a long time. It has proved its reliability. In this post I will show you how to create Layer 7 Load-Balancing. You can see below diagram on the Figure-1.

 

 

 

 

 

 

 

 

 

 

Configuration of the HaProxy(CentOS6):

Creating a Virtual Interface:

cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth0:0

Configure /etc/sysconfig/network-scripts/ifcfg-eth0:0

#[root@centos6 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0:0
DEVICE=eth0:0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
IPADDR=192.168.59.65
PREFIX=24
GATEWAY=192.168.59.2
DNS1=192.168.59.2
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0:0"

Restart network service

[root@centos6 haproxy]# service network restart
eth0      Link encap:Ethernet  HWaddr 00:0C:29:26:13:4F
          inet addr:192.168.59.60  Bcast:192.168.59.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe26:134f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41596 errors:0 dropped:0 overruns:0 frame:0
          TX packets:81978 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:11250635 (10.7 MiB)  TX bytes:9759307 (9.3 MiB)
eth0:0    Link encap:Ethernet  HWaddr 00:0C:29:26:13:4F
          inet addr:192.168.59.65  Bcast:192.168.59.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

Install HaProxy:

[root@centos6 ~]# yum install -y haproxy

Configure HaProxy: /etc/haproxy/haproxy.cfg

Copy below configuration at the end of the /etc/haproxy/haproxy.cfg file.

frontend front_myapp
        bind 192.168.59.65:80
        mode http
        acl url_blog1 path_beg /blog1
        acl url_blog2  path_beg /blog2
        use_backend back_blog1 if url_blog1
        use_backend back_blog2 if url_blog2
        default_backend back_default

backend back_blog1
        mode http
        server lubuntu 192.168.59.137:80 check

backend back_blog2
        mode http
        server lubuntu2 192.168.59.138:80 check

backend back_default
        mode http
        balance roundrobin
        server lubuntu 192.168.59.137:80 check
        server lubuntu2 192.168.59.138:80 check

listen stats  # Listen on localhost:9000
        bind 192.168.59.65:8080
        mode http
        stats enable  # Enable stats page
        stats hide-version  # Hide HAProxy version
        stats realm Haproxy\ Statistics  # Title text for popup window
        stats uri /hastats  # Stats URI
        #stats auth Username:Password  # Authentication

Check  configuration for syntax:

[root@centos6 haproxy]# haproxy -c -f haproxy.cfg
Configuration file is valid

Restart Service:

[root@centos6 haproxy]# service haproxy restart

For Statistics:

http://192.168.59.65:8080/hastats

 

 

 

 

 

Experiments:

Each request by client to the example.com will be redirected  lubuntu and  lubuntu2 (roundrobin) by the HaProxy.

 

 

 

 

 

 

Apache as a Forward Proxy

Proxy Server is a intermediary server that resides between client and  server. All requests originated by client send to the web server through  proxy server.  You can see below diagram with tcpdump analysis.

 

 

 

 

 

 

 

 

 

Configurations:

  1. Proxy Server Configurations(CentOS7)
  2. Install apache for using Proxy server.
[root@rhce ~]# yum install -y httpd

Create a file forward_proxy.conf under the /etc/httpd/conf.d and put the below lines into the configuration file.

Listen 7777
<VirtualHost 192.168.59.20:7777>
ProxyRequests On
ProxyVia On
ErrorLog logs/ex_error.log
CustomLog logs/ex_access.log combined
<ProxyMatch "http://example.com:8090/*">
Require ip 192.168.59.40
Require all denied
</ProxyMatch>
<ProxyMatch "http://example2.com:8090">
Require all denied
</ProxyMatch>
</VirtualHost>

Allow port 7777/tcp on the proxy server to the client make connections.

[root@rhce ~]#firewal-cmd --add-port=7777/tcp --permanent
[root@rhce ~]#firewal-cmd --reload

Restart httpd

[root@rhce ~]#systemctl restart httpd.service

2. Client Configurations

Setup the client to make request through proxy server to the 7777/tcp.

ansible@lubuntu:~$ export http_proxy=192.168.59.20:7777

Experiments:

Requst to example.com:8090. As you see in the debug message, request firstly goes to the proxy server.

ansible@lubuntu:~$ curl -v example.com:8090
* Rebuilt URL to: example.com:8090/
*   Trying 192.168.59.20...
* TCP_NODELAY set
* Connected to (nil) (192.168.59.20) port 7777 (#0)
> GET http://example.com:8090/ HTTP/1.1
> Host: example.com:8090
> User-Agent: curl/7.52.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Date: Sat, 19 Aug 2017 08:11:51 GMT
< Server: Apache/2.4.25 (Ubuntu)
< Last-Modified: Fri, 18 Aug 2017 08:36:39 GMT
< ETag: "19-55703071fac7d"
< Accept-Ranges: bytes
< Content-Length: 25
< Content-Type: text/html; charset=UTF-8
< Via: 1.1 rhce:8090
<
<h1>Hello example.com</>
* Curl_http_done: called premature == 0
* Connection #0 to host (nil) left intact

Request to example2.com

root@lubuntu:~# curl -v example2.com:8090
* Rebuilt URL to: example2.com:8090/
*   Trying 192.168.59.20...
* TCP_NODELAY set
* Connected to (nil) (192.168.59.20) port 7777 (#0)
> GET http://example2.com:8090/ HTTP/1.1
> Host: example2.com:8090
> User-Agent: curl/7.52.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 403 Forbidden
< Date: Sat, 19 Aug 2017 08:34:26 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.4.16
< Content-Length: 226
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access http://example2.com:8090/
on this server.</p>
</body></html>
* Curl_http_done: called premature == 0
* Connection #0 to host (nil) left intact

Logs on the Proxy Server.

192.168.59.40 - - [19/Aug/2017:10:30:42 +0200] "GET http://example.com:8090/ HTTP/1.1" 200 25 "-" "curl/7.52.1"
192.168.59.40 - - [19/Aug/2017:10:30:52 +0200] "GET http://example2.com:8090/ HTTP/1.1" 403 226 "-" "curl/7.52.1"
192.168.59.40 - - [19/Aug/2017:10:34:21 +0200] "GET http://example.com:8090/ HTTP/1.1" 200 25 "-" "curl/7.52.1"
192.168.59.40 - - [19/Aug/2017:10:34:26 +0200] "GET http://example2.com:8090/ HTTP/1.1" 403 226 "-" "curl/7.52.1"

SSH connection via Proxy Server:

I tried to connect to the host which is Web Server, using ssh protocol through the proxy server. But I got some errors. You can see below debug log, which was captured by ssh client. After some googling, I have found the solution to connect to the host via ssh. You can see below debug message before the proper solution.

...
debug1: key_load_public: No such file or directory
debug1: identity file /home/ansible/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Ubuntu-10https://manintheit.org/wp-admin/post.php?post=402&action=edit#
debug1: permanently_drop_suid: 1000
Proxy could not open connection to 192.168.59.50:  Proxy Error
ssh_exchange_identification: Connection closed by remote host

SSH Client Configuration to Connect Web Server Through Proxy Server.

Client Configuration:

You have two options for the client configuration to establish ssh connection through proxy.

Create file with the name config under the  ~/.ssh

Put the one of below configuration into the config file.
To use corkscrew you have to install this package.

Host *
    ProxyCommand corkscrew 192.168.59.20 7777 %h %p

OR

You can use nc tool.

Host *
    ProxyCommand nc -X connect -x 192.168.59.20:7777 %h %p

Proxy Server Configuration to allow SSH connection:

Proxy server by default allow http methods such as  GET, POST, HEAD. But for ssh connection we need to allow CONNECT request on the proxy server. To do that We put AllowCONNECT <port>  directive into the forward_proxy.conf file and restart the proxy server.

Listen 7777
<VirtualHost 192.168.59.20:7777>
  ProxyRequests On
  ProxyVia On
  AllowCONNECT 22           #Add this line to allow ssh connection.
  ErrorLog logs/ex_error.log
  CustomLog logs/ex_access.log combined
<ProxyMatch "http://example.com:8090/*">
        Require ip 192.168.59.40
        Require all denied
</ProxyMatch>
<ProxyMatch "http://example2.com:8090">
        Require all denied
</ProxyMatch>
</VirtualHost>

Experiments:

Gotcha !

We can able to connect through proxy.

debug1: Host '192.168.59.50' is known and matches the ECDSA host key.
debug1: Found key in /home/ansible/.ssh/known_hosts:2
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/ansible/.ssh/id_rsa
debug1: Trying private key: /home/ansible/.ssh/id_dsa
debug1: Trying private key: /home/ansible/.ssh/id_ecdsa
debug1: Trying private key: /home/ansible/.ssh/id_ed25519
debug1: Next authentication method: password
ansible@192.168.59.50's password:

Logs on the Proxy Server.[CONNECT] requests

192.168.59.40 - - [19/Aug/2017:11:29:36 +0200] "CONNECT 192.168.59.50:22 HTTP/1.0" 200 - "-" "-"
192.168.59.40 - - [19/Aug/2017:11:29:44 +0200] "CONNECT 192.168.59.50:22 HTTP/1.0" 200 - "-" "-"
192.168.59.40 - - [19/Aug/2017:11:31:25 +0200] "CONNECT 192.168.59.50:22 HTTP/1.0" 200 - "-" "-"

You can see iptables rule on the Web Server(192.168.59.50) below. It only allows connection from 192.168.59.20 which is proxy server. But we make actual requests from 192.168.59.40

target     prot opt source               destination
ACCEPT     tcp  --  192.168.59.20        anywhere             tcp dpt:8090
ACCEPT     tcp  --  192.168.59.20        anywhere             tcp dpt:ssh
REJECT     tcp  --  anywhere             anywhere             reject-with icmp-port-unreachable

 

 

 

 

 

 

Ansible Sample Scripts

Hello Folks!

In this post I will share you some playbooks. Actually, I wrote it before some purposes. You can change it and tweak it :).

1- Apache2 Virtual Host  deployment.

It deploys new Virtual Host configuration, enables configuration and checks syntax. If it is everything is all right then restarts an apache2, otherwise rollback.

#apache2-vhost.yml
---
  - hosts: lubuntu
    tasks:
      - name: copy vhost-lapiba.conf
        template: src=/home/ansible/ansible/templates/lapiba-vhost.conf dest=/etc/apache2/sites-available/lapiba-vhost.conf
        notify:
         - restart apache2
      - name: enable lapiba-vhost
        shell: /usr/sbin/a2ensite lapiba-vhost.conf
      - name: check lapiba-vhost configuration
        register: result
        shell: /usr/sbin/apache2ctl configtest
        ignore_errors: True
      - name: Disable-lapiba-vhost when error occurs
        shell: /usr/sbin/a2dissite lapiba-vhost.conf
        when: result|failed
      - name: remove lapiba-vhost
        file: dest=/etc/apache2/sites-available/lapiba-vhost.conf state=absent
        when: result|failed
      - name: Creates directory
        file: path=/var/www/lapiba state=directory
        when: result|success
      - name: put html files.
        template: src=/home/ansible/ansible/templates/index.html dest=/var/www/lapiba
        when: result|success
    handlers:
     - name: restart apache2
       service: name=apache2 state=restarted
       when: result|success

1.1 Apache Virtual Host Configuration

Minimalist Virtual Host configuration for testing purpose.

<VirtualHost *:80>
DocumentRoot "/var/www/lapiba"
ServerName lapiba
# Other directives here
</VirtualHost>

1.2 Sample index.html for lapiba

Sample index.html for testing virtual host. Do not forget to add labiba in your DNS  A record or /etc/hosts file.

<h1> Hello from LAPIBA<h1>
<h2>Ansible is everywhere</h2>

2 Sample Conditional Script

It pings google.com, if it cannot ping 10.10.10.1 in three times.

---
  - hosts: rhce
    tasks:
      - name: Echo
        shell: ping -c3 10.10.10.1
        register: result
        ignore_errors: True
      - name: Ping google.com
        shell: ping -c3 google.com
        when: result|failed
        #result|failed or success

3. Ansible Copy playbook 

It copies from localhost file /tmp/copy.me to remote host /tmp/hello.me

---
- hosts: rhce
  tasks:
    - name: Copying copy.me file to /tmp/hello.me
      copy: src=/tmp/copy.me dest=/tmp/hello.me

4. Install httpd on CentOS 7

It installs latest version of httpd, allows connections(temporary) to port 80 and starts the httpd process on the remote machine.

---
  - hosts: rhce
    tasks:
      - name: Install httpd
        yum: name=httpd state=latest
      - name: Copy index.html file to /var/www/html
        copy: src=/home/ansible/ansible/index.html dest=/var/www/html mode=0644 owner=root
        notify: restart httpd
      - name: open port 80
        shell: firewall-cmd --add-service=http
    handlers:
      - name: restart httpd
        service: name=httpd state=restarted

5. Replace the line 

It replaces the line GSSAPIAuthentication yes with #GSSAPIAuthentcation no on the remote host.

---
- hosts: rhce
  tasks:
    - replace:
        dest: /etc/ssh/gokay.me
        regexp: "GSSAPIAuthentication yes"
        replace: '#GSSAPIAuthentcation no'

6- Notify and Handlers

… handlers are only fired when certain tasks report changes, and are run at the end of each play:

---
  - hosts: lubuntu
    tasks:
      - name: "copy msg.html"
        template: src=/home/ansible/ansible/templates/template.html dest=/var/www/html/msg.html
        notify:
          - restart apache2
    handlers:
       - name: restart apache2
         service: name=apache2 state=restarted

7. Remove a file

It removes a file on the remote machine.

---
  - hosts: all
    tasks:
      - name: Removing hello.me
        file: dest  =/tmp/hello.me state=absent

8- Catching an Error with an Ansible

Something you suppose an error may not be an error for ansible. So, It is a good way to catch an error manually.  It will failed when no package found.(dadhaproxy in this case.)

---
  - hosts: rhce
    tasks:
      - name: search for package
        shell: yum search dadhaproxy
        register: result
        failed_when: "'No matches' in result.stdout"

9. Replace the line Granular way

It is granular way of replacing the lines. Lets assume that you have different flavors of Linux and each configuration different than other distribution. And you need an OS specific settings ? this playbooks for you, then.  🙂

---
#for ubuntu run it like below without ssh keys.
#ansible-playbook -l ubuntu  -i hosts playbooks/ssh_modify.yml -K -s
  - hosts: all
    tasks:
      - set_fact:
         sshd_ciphers: 'Pseudo Ciphers no ciphers'
         sshd_macs: 'CentOS MACs Pseudo'
        when: ansible_distribution=="CentOS"
      - set_fact:
         sshd_ciphers: 'Ciphers for ubuntu'
         sshd_macs: 'Ubuntu Macs Pseudo'
        when: ansible_distribution=="Ubuntu"
      - name: Update ciphers
        lineinfile:
          dest: /etc/ssh/sshd_template
          state: present
          regexp: '^Ciphers '
          line: 'Ciphers {{ hostvars[ansible_hostname]["sshd_ciphers"] }}'
        tags:
          - cip
      - name: Update MACs
        lineinfile:
          dest: /etc/ssh/sshd_template
          state: present
          regexp: '^MACs '
          line: 'MACs {{ hostvars[ansible_hostname]["sshd_macs"] }}'
        tags:
          - mac

10- Debugging 

Sometimes debugging is good way to troubleshoot an issue. Below debug saved my time. 🙂

---
  - hosts: lubuntu
    tasks:
      - name: check lapiba-vhost configuration
        shell: /usr/sbin/apache2ctl configtest
        register: result
        ignore_errors: True
      - debug:
          msg: Message "{{ result }}"
      - name: ping
        ping:
        when: result|failed

11- Validation with an Ansible

It is good way to validate configuration before, restarting service otherwise you can lock yourself out.

---
  - hosts: rhce
    tasks:
    - template:
        src: /home/ansible/ansible/templates/sshd_config
        dest: /etc/ssh/sshd_config
        owner: root
        group: root
        mode: '0600'
        validate: /usr/sbin/sshd -t -f %s
        backup: yes

 

12 Chek for Specific Update (YUM)

You can check specific RHEL advisory package and can update it.

---
        - hosts: all
          tasks:

                - name: "INFO Checking curl security vulnerabiliy"
                  shell: yum list-security --advisory RHSA-2017:0847
                  register: result
                  changed_when: "'RHSA-2017:0847' in result.stdout"
                  tags:
                   - check

                - name: "UPDATE curl security vulnerabiliy"
                  shell: yum update -y --advisory RHSA-2017:0847
                  tags:
                   - update

 

 

 

 

Syslog Server Configuration

Basic Syslog Server Configuration

In rsyslogd, different types of modules can be used:
■ Input modules: These are modules that have a name starting with im. Input
modules are used to specify from where rsyslogd will receive messages.
■ Output modules: These are modules that have a name starting with om. By
default, log messages are sent to the destinations as specified in /etc/rsyslog.
conf. By using output modules, messages can be sent elsewhere, like to a database or to the journal.
■ Other module types: Different other module types exist, such as parser modules, message modification modules, and more.

For more information download rsyslog-doc.

[root@rhce 07]# yum install rsyslog-doc

Minimal Configuration of Syslog Server :

#/etc/rsyslog.conf
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

$template DynFile, "/var/log/serverlogs/%$YEAR%/%$MONTH%/%HOSTNAME%.log"
*.* ?DynFile

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

Rsyslog can also send logs to the mysql database.  For this we need to import ommysql modules.

Sample configuration.
$ModLoad ommysql
*.* :ommysql:dbhost,dbname,dbuser,dbpassword

For more information about the rsyslog configuration, please visit gentoo.