Linux performance monitoring and fine-tuning
Understanding performance bottlenecks and fine-tuning them in a Linux system:
The first thing you must know in performance fine-tuning or Linux performance monitoring is the contributors. What resources contribute to the performance of applications?
If your are looking for system level performance fine-tuning, incase your server / PC is slow, this atricle still will help you to identify the applications / processes that are consuming your resources and contributing for performance degradation.
Typically, CPU, Memory, Disk (Input and Output) and Network. Note that, the network is not applicable for applications that run locally and doesn’t do Message Passing between them via the network.
Understanding the nature of applications:
Different applications that run on computers have different characteristics. The resource demands are different for various ones. We can classify the applications as below:
- CPU intensive or CPU hungry applications
- Memory intensive or Memory hungry applications
- CPU and Memory intensive or CPU and Memory hungry applications
- I/O intensive applications.
- Network-intensive applications.
The combinations can extend and mix and match. I am not listing all those here.
You must understand what is the nature of your application. Under which classification your application falls. Once you know, you have solved the problem half way.
How to identify the nature of applications or how to do Linux performance monitoring?
I don’t want to list all possible tools and confuse you. I will start with basic tools and show some case studies. This will help you to understand the concept in depth and master your skills in performance monitoring and fine-tuning.
| Resource | Monitoring tool |
| CPU | top |
| Memory | top, free, “/proc/<pid>/smaps”, vmstat |
| Disk (I/O) | iostat, top, lsof, iotop |
| Network | nethogs, iftop |
You must learn following “top” command hacks in order to monitor the performance of applications.
1. There are applications that run multi-threads. In “top” you just see a single process consuming CPU more than 100%. Notice PID 3263 in the following snapshot, it is consuming 195.3% CPU. Note that this is not a problem. It is the consolidated usage of all the child threads of that process.
# top -H
Above command option (H) will turn on threads mode in “top”. You can also do “top” first and then type “H” (upper case) to view all threads of processes.
2. If you have multiple cores in your system (which is common in all modern systems), you may have to monitor individual cores CPU utilisation. Do as follows:
Launch “top” utility and then type “1” (numeric one). You can see individual CPU core utilisation as follows. Notice “%Cpu0” and “%Cpu1” (I can see 2 cores since my system has 2 CPU cores). I have also enabled threads view. So you can now see CPU usage spread across 2 threads of parent process PID 3263.
Usecase for CPU intensive applications:
In the above snapshot notice that the program “eat_cpu_mem_p.o” is multithreaded and is CPU hungry. You can download and compile this program on your own from this article (see OpenMP threads in C).
You can also get it from GitHub.
How to address this problem?
Hmm..this is actually not a problem, it is just the nature of this application. You cant do much if you are an end user. You may consider migrating your application to a system with more CPU cores. If you are a developer you can optimize your code to reduce CPU usage. See Intel Developer Tools for performance monitoring and optimisation.
Usecase for I/O bound applications :
If your application is I/O bound and your system’s storage doesn’t have enough I/O bandwitdth, it sucks very badly.
In this case, your CPU will be busy doing nothing other than I/O wait. See the below snapshot to identify I/O wait “iowait” of applicaitons and overall system level I/O wait from top utility.
Note: I am updating this article.. stay tuned.