$ sed ‘/^$/d’ file.out > file1.out
ou
$ awk ‘NF>0’ file.out > file1.out
Linux – Recording your terminal with showterm
source: https://showterm.io/#use
Install
There are two ways to install showterm. The recommended way is to use ruby:
- gem install showterm
This works for all Mac users, and Linux users with ruby configured correctly.
If you are a Linux user who does not have ruby configured correctly, you can install showterm with:
- curl showterm.io/showterm > ~/bin/showterm
- chmod +x ~/bin/showterm
Use
If you have installed showterm, you can just run it:
- showterm [program to run]
If [program to run] is omitted it defaults to your shell (usually bash)
If you have not installed showterm, you can run the standalone version:
- bash <(curl record.showterm.io)
Linking
Each termshow gets its own link. You can add hash-fragments to customize playback,
- http://showterm.io/7b5f8d42ba021511e627e#slow — realtime
- http://showterm.io/7b5f8d42ba021511e627e#fast — double-speed
- http://showterm.io/7b5f8d42ba021511e627e#stop — stopped
access the site over https,
RED HAT – Change HostName in RHEL7
After installing the CentOS 7 on my server, i tried to change host name by modifying the /etc/sysconfig/network; it did not take an effect of the modification. Even after multiple reboot of server, the host name remains localhost.localdomain. The procedure to change the host name in CentOS 7 is now totally different from the previous version, this guide will help you to setup the host name on CentOS 7 / RHEL 7.
CentOS 7 supports three class of Host Names:
Static – The static host name is traditional host which can be chosen by the user and is stored in /etc/hostname file.
Transient – The transient host name is maintained by kernel and can be changed by DHCP and mDNS.
Pretty – It is a free form UTF -8 host name for the presentation to the user.
HostName can be,
- 64 character in a length
- Recommend to have FQDN
- Consists of a-z,A-Z, 0-9, “-”, “_” and “.” only
How to Change:
Before changing the host name, lets check the current host name.
[root@localhost ~]# hostname localhost.localdomain |
1. nmtui tool:
NetworkManaget tool is used to set the static host name in /etc/hostname file.
Set the host name.
restart the hostnamed to force the hostnamectl to notice the change in static host name.
[root@localhost ~]# systemctl restart systemd-hostnamed |
You can verify the change in host name.
[root@server ~]# hostname server.itzgeek.com [root@server ~]# cat /etc/hostname server.itzgeek.com [root@server ~]# cat /etc/sysconfig/network # Created by anaconda HOSTNAME=server.itzgeek.com |
2. hostnamectl:
hostnamectl is used to change the host name, with this tool we can change all the three class of host name; here we look only static host name.
Check the current host name.
[root@server ~]# hostnamectl status |
Static hostname: server.itzgeek.com Icon name: computer-vm Chassis: vm Machine ID: 565ea8b749544aca9d5563308f9e4bc2 Boot ID: 5c979d9b5f754df8b75a4e3aeabf2bad Virtualization: vmware Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.10.0-123.el7.x86_64 Architecture: x86_64 |
Set the hostname.
[root@server ~]# hostnamectl set-hostname client.itzgeek.com |
Check the host name again (Close the session and open new session using putty or console)
[root@client ~]# hostnamectl status |
Static hostname: client.itzgeek.com Icon name: computer-vm Chassis: vm Machine ID: 565ea8b749544aca9d5563308f9e4bc2 Boot ID: 5c979d9b5f754df8b75a4e3aeabf2bad Virtualization: vmware Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.10.0-123.el7.x86_64 Architecture: x86_64 |
If you use this command, you do not require to notify the change in host name. Close the current session and re launch the terminal.
3. nmcli tool:
It can be used to query and setup the static host name in /etc/hostname file.
Check the hostname.
[root@client ~]# nmcli general hostname client.itzgeek.com |
Change the host name.
[root@client ~]# nmcli general hostname server.itzgeek.com |
restart the hostnamed to force the hostnamectl to notice the change in static host name.
[root@client ~]# systemctl restart systemd-hostnamed |
4. Edit /etc/hostname
This is the simple, but requires a reboot of server to take an effect.
Note: Use the hostnamectl to change the host name, which fair better than other commands and does not require to update the kernel about the change in host name.
RED HAT – How to stop/start and disable/enable Firewall on Redhat 7
source: http://linuxconfig.org/how-to-stop-start-and-disable-enable-firewall-on-redhat-7-linux-system
The firewall on Redhat 7 Linux system is enabled by default. Normally there should not be a need to disable firewall but it may be quite handy for testing purposes etc. On Redhat 7 Linux system the firewall run as firewalld daemon. Bellow command can be used to check the firewall status:
[root@rhel7 ~]# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
Active: active (running) since Thu 2014-09-04 19:18:47 EST; 3 months 28 days ago
Main PID: 539 (firewalld)
CGroup: /system.slice/firewalld.service
└─539 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
Sep 04 19:18:45 rhel7 systemd[1]: Starting firewalld - dynamic firewall daemon...
Sep 04 19:18:47 rhel7 systemd[1]: Started firewalld - dynamic firewall daemon.
From the above output we can see that the firewall is enabled, which means it will start automatically after reboot and that is also current active. Furthermore, or you can even check all currently applied rules with:
[root@rhel7 ~]# iptables-save
1. Stop and Start RHEL7 firewall
The firewall on Redhat 7 Linux system can be stopped by a following command:
[root@rhel7 ~]# service firewalld stop Redirecting to /bin/systemctl stop firewalld.service
Stopped firewall will start again after system’s reboot. To start firewall on Redhat 7 Linux system use:
[root@rhel7 ~]# service firewalld start Redirecting to /bin/systemctl start firewalld.service
2. Disable and Enable RHEL7 firewall
In order to completely disable RHEL7 firewall so it would no load after reboot run:
[root@rhel7 ~]# systemctl disable firewalld rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service' rm '/etc/systemd/system/basic.target.wants/firewalld.service'
Now the firewall would not start after system’s reboot. To enable the firewall again run:
[root@rhel7 ~]# systemctl enable firewalld ln -s '/usr/lib/systemd/system/firewalld.service' '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service' ln -s '/usr/lib/systemd/system/firewalld.service' '/etc/systemd/system/basic.target.wants/firewalld.service'
Open files per user (nofiles)
How to verify open files per user:
lsof | grep <user> | awk ‘{print $NF}’ | wc -l
AIX – How to find processes have listening on ports
LSOF:
$ lsof -i :50000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
db2sysc 4128774 db2inst1 5u IPv6 0xf1000e00019f3bb8 0t0 TCP *:50000 (LISTEN)
NETSTAT:
# netstat -Aan | grep LISTEN | grep 50000
f1000e00019f3bb8 tcp4 0 0 *.50000 *.* LISTEN
How to remove this process:
# rmsock f1000e00019f3bb8 tcpcb
The socket 0xf1000e00019f3bb8 is being held by proccess 4128774(writesrv).
RED HAT – Bash Remote Code Execution (Shellshock) on RED HAT 4
Pessoal,
Para quem possui ambientes em Back Level, raramente irá encontrar pacotes com a solução para a vulnerabilidade de execução de códigos remotos através do bash. No meu caso, tenho diversos servidores com RED HAT 4 (em diversos níveis de patch), devido aos pré-requisitos das aplicações. Como não possuo o suporte estendido da RED HAT, tive que procurar outra forma de poder sanar este bug.
Segue o que fiz:
-> Teste de vulnerabilidade, caso apareça a palavra “vulnerable”, quer dizer que a versão do seu bash possui a vulnerabilidade informada:
env ‘x=() { :;}; echo vulnerable’ ‘BASH_FUNC_x()=() { :;}; echo vulnerable’ bash -c “echo test”
-> Baixe o pacote do repositório publico do Oracle Linux:
-> Envie o pacote para o servidor com a vulnerabilidade e atualize o pacote do bash:
rpm -Uvh bash-3.0-27.0.3.el4.x86_64.rpm
-> Execute novamente o teste de vulnerabilidade, o resultado deve ser semelhante a este:
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `BASH_FUNC_x’
test
RED HAT – Bash Code Injection Vulnerability via Specially Crafted Environment Variables (CVE-2014-6271, CVE-2014-7169)
source: https://access.redhat.com/articles/1200223
Red Hat has been made aware of a vulnerability affecting all versions of the bash package as shipped with Red Hat products. This vulnerability CVE-2014-6271 could allow for arbitrary code execution. Certain services and applications allow remote unauthenticated attackers to provide environment variables, allowing them to exploit this issue.
Update: 2014-09-26 05:15 UTC
Red Hat has become aware that the patch for CVE-2014-6271 is incomplete. An attacker can provide specially-crafted environment variables containing arbitrary commands that will be executed on vulnerable systems under certain conditions. The new issue has been assigned CVE-2014-7169.
Updated bash packages that address CVE-2014-7169 are now available for Red Hat Enterprise Linux 5, 6, and 7, Red Hat Enterprise Linux 4 Extended Life Cycle Support, Red Hat Enterprise Linux 5.6 Long Life, Red Hat Enterprise Linux 5.9 Extended Update Support, Red Hat Enterprise Linux 6.2 Advanced Update Support, and Red Hat Enterprise Linux 6.4 Extended Update Support, and Shift_JIS for Red Hat Enterprise Linux 5 and 6. See also Resolution for Bash Code Injection Vulnerability via Specially Crafted Environment Variables (CVE-2014-6271, CVE-2014-7169) in Red Hat Enterprise Linux.
Diagnostic Steps
Red Hat Access Labs has provided a script to help confirm if a system is patched against to the Shellshock vulnerability. You can also manually test your version of Bash by running the following command:
$ env 'x=() { :;}; echo vulnerable' 'BASH_FUNC_x()=() { :;}; echo vulnerable' bash -c "echo test"
If the output of the above command contains a line containing only the word vulnerable you are using a vulnerable version of Bash. The patch used to fix this issue ensures that no code is allowed after the end of a Bash function.
Note that different Bash versions will also print different warnings while executing the above command. The Bash versions without any fix produce the following output:
$ env 'x=() { :;}; echo vulnerable' 'BASH_FUNC_x()=() { :;}; echo vulnerable' bash -c "echo test"
vulnerable
bash: BASH_FUNC_x(): line 0: syntax error near unexpected token `)'
bash: BASH_FUNC_x(): line 0: `BASH_FUNC_x() () { :;}; echo vulnerable'
bash: error importing function definition for `BASH_FUNC_x'
test
The versions with only the original CVE-2014-6271 fix applied produce the following output:
$ env 'x=() { :;}; echo vulnerable' 'BASH_FUNC_x()=() { :;}; echo vulnerable' bash -c "echo test"
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
bash: error importing function definition for `BASH_FUNC_x()'
test
The versions with additional fixes from RHSA-2014:1306, RHSA-2014:1311 and RHSA-2014:1312 produce the following output:
$ env 'x=() { :;}; echo vulnerable' 'BASH_FUNC_x()=() { :;}; echo vulnerable' bash -c "echo test"
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `BASH_FUNC_x'
test
The difference in the output is caused by additional function processing changes explained in the “How does this impact systems” section below.
The fix for CVE-2014-7169 ensures that the system is protected from the file creation issue. To test if your version of Bash is vulnerable to CVE-2014-7169, run the following command:
$ cd /tmp; rm -f /tmp/echo; env 'x=() { (a)=>\' bash -c "echo date"; cat /tmp/echo
bash: x: line 1: syntax error near unexpected token `='
bash: x: line 1: `'
bash: error importing function definition for `x'
Fri Sep 26 11:49:58 GMT 2014
If your system is vulnerable, the time and date information will be output on the screen and a file called /tmp/echo will be created.
If your system is not vulnerable, you will see output similar to:
$ cd /tmp; rm -f /tmp/echo; env 'x=() { (a)=>\' bash -c "echo date"; cat /tmp/echo
date
cat: /tmp/echo: No such file or directory
If your system is vulnerable, you can fix these issues by updating to the most recent version of the Bash package by running the following command:
# yum update bash
How does this impact systems
This issue affects all products which use the Bash shell and parse values of environment variables. This issue is especially dangerous as there are many possible ways Bash can be called by an application. Quite often if an application executes another binary, Bash is invoked to accomplish this. Because of the pervasive use of the Bash shell, this issue is quite serious and should be treated as such.
All versions prior to those listed as updates for this issue are vulnerable to some degree.
See the appropriate remediation article for specifics.
The patch for CVE-2014-7169 introduces changes to how Bash evaluates environment variables. Applications which directly create Bash functions as environment variables need to be made aware of these changes. Previously, a function had to be stored in an environment variable of the same name. For example, the function “compute” would be stored in an environment variable named “compute”. With the patch for CVE-2014-7169 applied, it would need to use the name “BASH_FUNC_compute()”. As a result, there are now two pairs of parentheses in the environment string, as in “BASH_FUNC_compute()=() { }”.
Functions written in Bash itself do not need to be changed, even if they are exported with “export -f”. Bash will transparently apply the appropriate naming when exporting, and reverse the process when importing function definitions.
Services that create such environment variables will need to be restarted to work with the new version of Bash. This behavior is not used by any of the packages provided in any version of Red Hat Enterprise Linux.
Products Affected:
| Product/Channel | Fixed in package | Remediation details |
|---|---|---|
| Red Hat Enterprise Linux 7 | bash-4.2.45-5.el7_0.4 | Red Hat Enterprise Linux |
| Red Hat Enterprise Linux 6 | bash-4.1.2-15.el6_5.2 | Red Hat Enterprise Linux |
| bash-4.1.2-15.el6_5.1.sjis.2 | Red Hat Enterprise Linux | |
| bash-4.1.2-9.el6_2.2 | Red Hat Enterprise Linux 6.2 AUS | |
| bash-4.1.2-15.el6_4.2 | Red Hat Enterprise Linux 6.4 EUS | |
| Red Hat Enterprise Linux 5 | bash-3.2-33.el5_11.4 | Red Hat Enterprise Linux |
| bash-3.2-33.el5_11.1.sjis.2 | Red Hat Enterprise Linux | |
| bash-3.2-24.el5_6.2 | Red Hat Enterprise Linux 5.6 LL | |
| bash-3.2-32.el5_9.3 | Red Hat Enterprise Linux 5.9 EUS | |
| Red Hat Enterprise Linux 4 | bash-3.0-27.el4.4 | Red Hat Enterprise Linux 4 ELS |
Common Configuration Examples:
Red Hat performed an analysis to better understand the magnitude of this issue and how it affects various configurations. The below list is not exhaustive, but is meant to give some examples of how this issue affects certain configurations, and why the high level of complexity makes it impossible to specify something is not affected by this issue. The best course of action is to upgrade Bash to a fixed version.
| Package | Description |
|---|---|
| httpd | CGI scripts are likely affected by this issue: when a CGI script is run by the web server, it uses environment variables to pass data to the script. These environment variables can be controlled by the attacker. If the CGI script calls Bash, the script could execute arbitrary code as the httpd user. mod_php, mod_perl, and mod_python do not use environment variables and we believe they are not affected. |
| Secure Shell (SSH) | It is not uncommon to restrict remote commands that a user can run via SSH, such as rsync or git. In these instances, this issue can be used to execute any command, not just the restricted command. |
| dhclient | The Dynamic Host Configuration Protocol Client (dhclient) is used to automatically obtain network configuration information via DHCP. This client uses various environment variables and runs Bash to configure the network interface. Connecting to a malicious DHCP server could allow an attacker to run arbitrary code on the client machine. |
| CUPS | It is believed that CUPS is affected by this issue. Various user supplied values are stored in environment variables when cups filters are executed. |
| sudo | Commands run via sudo are not affected by this issue. Sudo specifically looks for environment variables that are also functions. It could still be possible for the running command to set an environment variable that could cause a Bash child process to execute arbitrary code. |
| Firefox | We do not believe Firefox can be forced to set an environment variable in a manner that would allow Bash to run arbitrary commands. It is still advisable to upgrade Bash as it is common to install various plug-ins and extensions that could allow this behavior. |
| Postfix | The Postfix server will replace various characters with a ?. While the Postfix server does call Bash in a variety of ways, we do not believe an arbitrary environment variable can be set by the server. It is however possible that a filter could set environment variables. |
A more detailed analysis of the flaw is available at: https://securityblog.redhat.com/2014/09/24/bash-specially-crafted-environment-variables-code-injection-attack
Frequently Asked Questions
This FAQ is for the vulnerability CVE-2014-6271 in Bash.
Some additional information regarding CVE-2014-6271 and CVE-2014-7169 is available at: https://securityblog.redhat.com/2014/09/26/frequently-asked-questions-about-the-shellshock-bash-flaws/
I believe my system may have been compromised due to this vulnerability, what should I do?
If you have run the diagnostic steps in this article, and your system still appears to be vulnerable, or you believe your system has been compromised, open a support case with Red Hat or contact Red Hat support by phone.
Do I need to reboot or restart services after installing this update?
No, a reboot of your system or any of your services is not required. This vulnerability is in the initial import of the process environment from the kernel. This only happens when Bash is started. After the update that fixes this issue is installed, such new processes will use the new code, and will not be vulnerable. Conversely, old processes will not be started again, so the vulnerability does not materialize. If you have a strong reason to suspect that a system was compromised by this vulnerability then a system reboot should be performed after the update is installed as a best security practice and security checks should be analyzed for suspicious activity.
Are other shells vulnerable to this issue?
Red Hat has tested other shells for this issue. We could not reproduce the behavior seen in Bash. If similar issues are discovered in other shells we will release updates as appropriate.
Are there any possible mitigations against this issue?
Please see Mitigating the shellshock vulnerability (CVE-2014-6271 and CVE-2014-7169) for details on potential mitigations.
Linux – CPU Wait Problem (IO)
source: http://www.chileoffshore.com/en/interesting-articles/126-linux-wait-io-problem (This post is very good!)
Detection of Problem
When you log into a Linux box, if the WA is present and with very high percentage, you will feel the login process will take much longer time than the normal. Then any operation will also take much longer than they usually do.
Determination of Problem
To determine if the problem is caused by wait io is relatively easy, there are several commands can be used, one of them is the command vmstat:
# vmstat 2 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 1 1680 5134528 132148 165904 0 0 0 19 4 0 0 0 72 27 0 0 0 1680 5134528 132148 165904 0 0 0 14 59 88 0 0 74 26 0 0 0 1680 5134528 132148 165904 0 0 0 0 49 74 0 0 100 0 0 0 0 1680 5134528 132148 165916 0 0 0 32 48 82 0 0 44 57 0 0 0 1680 5134528 132148 165916 0 0 0 0 53 79 0 0 100 0 0 0 2 1680 5134528 132148 165924 0 0 0 28 58 89 0 0 58 42 0 0 0 1680 5134528 132148 165924 0 0 0 2 40 72 0 0 95 6 0 0 0 1680 5134528 132148 165924 0 0 0 0 49 74 0 0 100 0 0
From here you can see the pre-last column is the “wa”, it is always with some value there and it means your system is constantly waiting IO operations. So, your Linux Box will never works well.
Please note, in this result, the sum of columns id (Idle) and wa (Wait IO) is almost 100, this means there are some configuration problems. Because the machine is seem it does not doing any thing but disk IO operations. It typically is caused by some kind of disk configuration problem, like journal for ext4.
The vmstat command shows you both foreground and background processes. So, it is not very precise to find out if there is a real IO problem. For example, if some user is restoring a specific file from a multi-volumns tar backup, it will cause the WA very high, but it is a foreground process which is not blocking whole linux server. But it is really telling you that some process is causing high IO usage.
The Cause
One of most possible reason which causes the problem is the “Journal Flushing Operation”. It periodically flushes journal commits and other modifications to disk. To determine if it is the cause, using this command:
# ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 2 0.0 0.0 0 0 ? S May22 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S May22 0:00 \_ [migration/0] root 4 0.0 0.0 0 0 ? S May22 0:00 \_ [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S May22 0:00 \_ [watchdog/0] root 6 0.0 0.0 0 0 ? S May22 0:50 \_ [events/0] root 7 0.0 0.0 0 0 ? S May22 0:00 \_ [cpuset] root 8 0.0 0.0 0 0 ? S May22 0:00 \_ [khelper] root 9 0.0 0.0 0 0 ? S May22 0:00 \_ [netns] root 10 0.0 0.0 0 0 ? S May22 0:00 \_ [async/mgr] root 11 0.0 0.0 0 0 ? S May22 0:00 \_ [pm] ...
Here the most important column for us is the STAT, which means some thing as follow:
D Uninterruptible sleep (usually IO)
R Running or runnable (on run queue)
S Interruptible sleep (waiting for an event to complete)
T Stopped, either by a job control signal or because it is being traced.
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z Defunct ("zombie") process, terminated but not reaped by its parent.
So, just as mentioned above, if a process with its stat with “D”, it means it is actually taking all CPU resource with no any possible interruption. This means your Linux Box will wait on IO and does not responding any other commands if such process is always there.
To nail down which process is “eating” your CPU time, you can use this command which samples all process with D flag in every second:
# while true; do date; ps auxf | awk '{if($8=="D") print $0;}'; sleep 1; done
Tue Aug 23 20:03:42 CLT 2011
Tue Aug 23 20:03:43 CLT 2011
root 321 0.0 0.0 0 0 ? D May22 4:11 \_ [jbd2/dm-0-8]
Tue Aug 23 20:03:44 CLT 2011
Tue Aug 23 20:03:45 CLT 2011
Tue Aug 23 20:03:46 CLT 2011
...
Tue Aug 23 20:03:47 CLT 2011
Tue Aug 23 20:03:53 CLT 2011
Tue Aug 23 20:03:54 CLT 2011
root 302 0.0 0.0 0 0 ? D May22 2:58 \_ [kdmflush]
root 321 0.0 0.0 0 0 ? D May22 4:11 \_ [jbd2/dm-0-8]
Tue Aug 23 20:03:55 CLT 2011
Tue Aug 23 20:03:56 CLT 2011
Tue Aug 23 20:03:57 CLT 2011
Tue Aug 23 20:03:58 CLT 2011
Tue Aug 23 20:03:59 CLT 2011
root 302 0.0 0.0 0 0 ? D May22 2:58 \_ [kdmflush]
root 321 0.0 0.0 0 0 ? D May22 4:11 \_ [jbd2/dm-0-8]
Tue Aug 23 20:04:00 CLT 2011
Tue Aug 23 20:04:01 CLT 2011
Tue Aug 23 20:04:02 CLT 2011
From the result above, you see there are two process which are consuming your CPU with Wait IO, kdmflush and jbd2/dm-0-8
Note: if only date/time are displayed on the screen, it means there is no any serious WaitIO problem there.
Also you can use the following command to realize a monitoring on these two processes:
# while true; do ps auxf | grep D | grep -E "(jbd2\/dm\.*|kdmflush)"; sleep 1; done root 302 0.0 0.0 0 0 ? D May22 2:58 \_ [kdmflush] root 321 0.0 0.0 0 0 ? D May22 4:11 \_ [jbd2/dm-0-8] root 321 0.0 0.0 0 0 ? D May22 4:11 \_ [jbd2/dm-0-8] root 321 0.0 0.0 0 0 ? D May22 4:11 \_ [jbd2/dm-0-8] root 321 0.0 0.0 0 0 ? D May22 4:11 \_ [jbd2/dm-0-8] root 302 0.0 0.0 0 0 ? D May22 2:58 \_ [kdmflush] root 321 0.0 0.0 0 0 ? D May22 4:11 \_ [jbd2/dm-0-8] root 302 0.0 0.0 0 0 ? D May22 2:58 \_ [kdmflush] root 321 0.0 0.0 0 0 ? D May22 4:11 \_ [jbd2/dm-0-8]
As you can see, these two processes are responsible for Wait IO of your linux server.
Solution
First of all, the reason of high WA is not always the same. But the solution will always on those processes which are with STAT as D. In this case, the configuration of “Journal Disk” should be reconsidered. If the server is a machine for development, it is not recommended to use Journal to protect the hard disk. If the server is a product server, some kind of RAID should be used to protect the failure of disks.
Others:
watch -n 1 “(ps aux | awk ‘\$8 ~ /D/ { print \$0 }’)”
Linux – How to copy skel for home directory
cp -r /etc/skel/.[^.]* /home/user
Security Information Fans 