前言
NAME
pidstat - Report statistics for Linux tasks.
vmstat用来观测系统整体的性能情况，并不能观测单个进程，使用pidstat来观测单个进程的性能情况。
一、pidstat简介
pidstat命令用于监视当前由Linux内核管理的各个任务，使用 选项 -p 将指定的任务的信息输出到标准输出中，或者使用选项 -p ALL将Linux内核管理的每个任务的信息输出到标准输出中。若不指定任何任务等同于使用选项 -p ALL，但只有活动任务(统计值非零的任务)将出现在报告中。
pidstat命令还可以用于监视所选任务的子进程，参考选项 -T。
interval参数指定每个报告之间以秒为单位的时间间隔。值为0(或者根本没有参数)表示将报告自系统启动(boot)以来的一段时间内的任务统计信息。如果没有将interval参数设置为零，则可以将count参数与interval参数一起指定。count的值决定了间隔几秒生成的报告的数量。如果指定interval参数而不带count参数，pidstat命令将连续生成报表。
pidstat ...... [ interval[ count ] ]
每隔2秒生成一份任务统计信息，总共生成5次任务统计信息，同时还会产生一个该时间段内的平均任务统计信息。
[root@localhost ~]# pidstat 2 5
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain) 11/24/2022 _x86_64_ (4 CPU)
03:45:25 PM UID PID %usr %system %guest %CPU CPU Command
03:45:27 PM 0 4524 5.97 1.99 0.00 7.96 0 accounts-daemon
03:45:27 PM 998 4533 6.97 2.99 0.00 9.95 0 polkitd
03:45:27 PM 81 4541 6.97 1.49 0.00 8.46 1 dbus-daemon
03:45:27 PM 0 14329 0.00 0.50 0.00 0.50 3 bash
03:45:27 PM 0 15031 0.00 0.50 0.00 0.50 0 pidstat
03:45:27 PM 0 21424 1.99 0.50 0.00 2.49 3 gsd-account
03:45:27 PM UID PID %usr %system %guest %CPU CPU Command
03:45:29 PM 0 482 0.50 0.00 0.00 0.50 2 sshd
03:45:29 PM 0 4524 5.00 2.00 0.00 7.00 2 accounts-daemon
03:45:29 PM 998 4533 8.50 2.50 0.00 11.00 3 polkitd
03:45:29 PM 81 4541 7.00 2.00 0.00 9.00 0 dbus-daemon
03:45:29 PM 0 15031 0.00 0.50 0.00 0.50 0 pidstat
03:45:29 PM 0 21424 2.00 1.00 0.00 3.00 3 gsd-account
03:45:29 PM UID PID %usr %system %guest %CPU CPU Command
03:45:31 PM 0 9 0.00 0.50 0.00 0.50 3 rcu_sched
03:45:31 PM 0 4524 6.50 1.00 0.00 7.50 0 accounts-daemon
03:45:31 PM 998 4533 7.00 3.50 0.00 10.50 3 polkitd
03:45:31 PM 81 4541 7.00 1.00 0.00 8.00 1 dbus-daemon
03:45:31 PM 0 15031 0.50 0.50 0.00 1.00 0 pidstat
03:45:31 PM 0 21424 2.50 0.00 0.00 2.50 3 gsd-account
03:45:31 PM UID PID %usr %system %guest %CPU CPU Command
03:45:33 PM 0 482 0.00 0.50 0.00 0.50 2 sshd
03:45:33 PM 0 4524 5.47 1.99 0.00 7.46 0 accounts-daemon
03:45:33 PM 998 4533 7.46 3.48 0.00 10.95 0 polkitd
03:45:33 PM 81 4541 6.47 1.99 0.00 8.46 1 dbus-daemon
03:45:33 PM 0 14329 0.50 0.50 0.00 1.00 3 bash
03:45:33 PM 0 15031 0.00 0.50 0.00 0.50 0 pidstat
03:45:33 PM 0 21086 0.50 0.00 0.00 0.50 1 gvfsd-fuse
03:45:33 PM 0 21424 2.49 1.00 0.00 3.48 3 gsd-account
03:45:33 PM 0 21440 0.50 0.00 0.00 0.50 2 gsd-color
03:45:33 PM UID PID %usr %system %guest %CPU CPU Command
03:45:35 PM 0 4524 6.00 1.00 0.00 7.00 3 accounts-daemon
03:45:35 PM 998 4533 8.50 2.00 0.00 10.50 1 polkitd
03:45:35 PM 81 4541 6.50 1.50 0.00 8.00 2 dbus-daemon
03:45:35 PM 0 6183 0.00 0.50 0.00 0.50 3 pcscd
03:45:35 PM 0 15031 0.50 0.50 0.00 1.00 0 pidstat
03:45:35 PM 0 21423 0.00 0.50 0.00 0.50 1 gsd-smartcard
03:45:35 PM 0 21424 2.00 0.50 0.00 2.50 0 gsd-account
03:45:35 PM 0 21805 0.50 0.00 0.00 0.50 3 fwupd
Average: UID PID %usr %system %guest %CPU CPU Command
Average: 0 9 0.00 0.10 0.00 0.10 - rcu_sched
Average: 0 482 0.10 0.10 0.00 0.20 - sshd
Average: 0 4524 5.79 1.60 0.00 7.39 - accounts-daemon
Average: 998 4533 7.68 2.89 0.00 10.58 - polkitd
Average: 81 4541 6.79 1.60 0.00 8.38 - dbus-daemon
Average: 0 6183 0.00 0.10 0.00 0.10 - pcscd
Average: 0 14329 0.10 0.20 0.00 0.30 - bash
Average: 0 15031 0.20 0.50 0.00 0.70 - pidstat
Average: 0 21086 0.10 0.00 0.00 0.10 - gvfsd-fuse
Average: 0 21423 0.00 0.10 0.00 0.10 - gsd-smartcard
Average: 0 21424 2.20 0.60 0.00 2.79 - gsd-account
Average: 0 21440 0.10 0.00 0.00 0.10 - gsd-color
Average: 0 21805 0.10 0.00 0.00 0.10 - fwupd
二、pidstat 的使用
2.1 pidstat -C comm
只显示命令名中包含字符串comm的任务。该字符串可以是正则表达式。
[root@localhost ~]# pidstat -C sshd
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain) 11/24/2022 _x86_64_ (4 CPU)
03:54:27 PM UID PID %usr %system %guest %CPU CPU Command
03:54:27 PM 0 356 0.00 0.00 0.00 0.00 0 sshd
03:54:27 PM 0 360 0.00 0.00 0.00 0.00 2 sshd
03:54:27 PM 0 482 0.00 0.00 0.00 0.00 3 sshd
03:54:27 PM 0 486 0.00 0.00 0.00 0.00 0 sshd
03:54:27 PM 0 5174 0.00 0.00 0.00 0.00 1 sshd
2.2 pidstat -d
-d Report I/O statistics
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain) 11/24/2022 _x86_64_ (4 CPU)
04:02:11 PM UID PID kB_rd/s kB_wr/s kB_ccwr/s Command
04:02:11 PM 0 1 0.96 1.64 0.13 systemd
04:02:11 PM 0 365 0.00 0.00 0.00 bash
04:02:11 PM 0 490 0.00 0.00 0.00 bash
04:02:11 PM 0 2173 0.00 0.00 0.00 xfsaild/dm-0
04:02:11 PM 0 2254 0.00 0.00 0.00 systemd-journal
04:02:11 PM 0 2276 0.00 0.00 0.00 lvmetad
04:02:11 PM 0 2297 0.02 0.00 0.00 systemd-udevd
04:02:11 PM 0 4496 0.00 0.02 0.00 auditd
04:02:11 PM 0 4524 0.00 0.00 0.00 accounts-daemon
04:02:11 PM 172 4527 0.00 0.00 0.00 rtkit-daemon
04:02:11 PM 0 4528 0.00 0.00 0.00 alsactl
04:02:11 PM 998 4533 0.00 0.00 0.00 polkitd
04:02:11 PM 70 4539 0.00 0.00 0.00 avahi-daemon
04:02:11 PM 0 4540 0.00 0.00 0.00 bluetoothd
04:02:11 PM 81 4541 0.00 0.00 0.00 dbus-daemon
04:02:11 PM 0 4549 0.00 0.00 0.00 udisksd
04:02:11 PM 0 4551 0.00 0.00 0.00 abrtd
04:02:11 PM 0 4554 0.00 0.00 0.00 abrt-watch-log
04:02:11 PM 994 4561 0.00 0.00 0.00 lsmd
04:02:11 PM 0 4591 0.00 0.00 0.00 gssproxy
04:02:11 PM 0 4635 0.00 0.00 0.00 firewalld
......
UID：正在监视的任务的真实用户标识号。
PID：正在监视的任务的标识号。
kB_rd/s：任务每秒从磁盘读取的 kilobytes。
kB_wr/s：任务每秒已写入或将写入磁盘的 kilobytes。
kB_ccwr/s：任务已经取消写入磁盘的 kilobytes，这可能发生在任务截断某些脏页缓存时（dirty pagecache：缓存中的数据还没有写入磁盘）。
Command：任务的命令名称。
2.3 pidstat -l
-l Display the process command name and all its arguments.
显示任务的完整路径和任务执行时的参数，和从/proc/fd/cmdline 中的值一样
04:25:21 PM UID PID %usr %system %guest %CPU CPU Command
......
04:25:21 PM 0 614 0.00 0.00 0.00 0.00 1 kworker/u8:0
04:25:21 PM 0 1463 0.00 0.00 0.00 0.00 2 i915/signal:0
04:25:21 PM 0 2173 0.00 0.01 0.00 0.01 3 xfsaild/dm-0
04:25:21 PM 0 2174 0.00 0.00 0.00 0.00 0 kworker/0:1H
04:25:21 PM 0 2254 0.00 0.00 0.00 0.00 1 /usr/lib/systemd/systemd-journald
04:25:21 PM 0 2276 0.00 0.00 0.00 0.00 2 /usr/sbin/lvmetad -f
04:25:21 PM 0 2295 0.00 0.00 0.00 0.00 3 kworker/3:1H
04:25:21 PM 0 2297 0.00 0.00 0.00 0.00 3 /usr/lib/systemd/systemd-udevd
04:25:21 PM 0 4428 0.00 0.00 0.00 0.00 1 xfsaild/sda2
04:25:21 PM 0 4496 0.00 0.00 0.00 0.00 2 /sbin/auditd
04:25:21 PM 0 4498 0.00 0.00 0.00 0.00 0 /sbin/audispd
04:25:21 PM 0 4499 0.00 0.00 0.00 0.00 2 /usr/sbin/sedispatch
04:25:21 PM 32 4522 0.00 0.00 0.00 0.00 3 /sbin/rpcbind -w
04:25:21 PM 0 4523 0.00 0.00 0.00 0.00 1 /usr/lib/systemd/systemd-logind
04:25:21 PM 0 4524 2.26 0.60 0.00 2.86 0 /usr/libexec/accounts-daemon
04:25:21 PM 172 4527 0.00 0.00 0.00 0.00 3 /usr/libexec/rtkit-daemon
04:25:21 PM 0 4528 0.00 0.00 0.00 0.00 0 /usr/sbin/alsactl -s -n 19 -c -E ALSA_CONFIG_PATH=/etc/alsa/alsactl.conf --initfile=/lib/alsa/init/00main rdaemon
04:25:21 PM 0 4529 0.00 0.00 0.00 0.00 0 /sbin/rngd -f
.......
数据来源就是 /proc/fd/stat，/proc/fd/cmdline，/proc/fd/status：
[root@localhost ~]# strace -e trace=open pidstat -l
......
open("/proc/1/stat", O_RDONLY) = 4
open("/proc/1/cmdline", O_RDONLY) = 4
open("/proc/1/status", O_RDONLY) = 4
open("/proc/2/stat", O_RDONLY) = 4
open("/proc/2/cmdline", O_RDONLY) = 4
open("/proc/2/status", O_RDONLY) = 4
open("/proc/3/stat", O_RDONLY) = 4
open("/proc/3/cmdline", O_RDONLY) = 4
open("/proc/3/status", O_RDONLY) = 4
open("/proc/5/stat", O_RDONLY) = 4
open("/proc/5/cmdline", O_RDONLY) = 4
open("/proc/5/status", O_RDONLY) = 4
open("/proc/7/stat", O_RDONLY) = 4
open("/proc/7/cmdline", O_RDONLY) = 4
open("/proc/7/status", O_RDONLY) = 4
open("/proc/8/stat", O_RDONLY) = 4
open("/proc/8/cmdline", O_RDONLY) = 4
open("/proc/8/status", O_RDONLY) = 4
......
2.4 pidstat -p
查看指定进程的信息统计：
[root@localhost ~]# pidstat -p `pidof top`
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain) 11/24/2022 _x86_64_ (4 CPU)
04:36:10 PM UID PID %usr %system %guest %CPU CPU Command
04:36:10 PM 0 4121 0.00 0.00 0.00 0.00 2 top
查看 pidstat 进程本身信息统计：
[root@localhost ~]# pidstat -p SELF
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain) 11/24/2022 _x86_64_ (4 CPU)
04:36:41 PM UID PID %usr %system %guest %CPU CPU Command
04:36:41 PM 0 5340 0.00 0.00 0.00 0.00 3 pidstat
查看所有进程信息统计：
[root@localhost ~]# pidstat -p ALL
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain) 11/24/2022 _x86_64_ (4 CPU)
04:37:28 PM UID PID %usr %system %guest %CPU CPU Command
04:37:28 PM 0 1 0.01 0.01 0.00 0.01 2 systemd
04:37:28 PM 0 2 0.00 0.00 0.00 0.00 0 kthreadd
04:37:28 PM 0 3 0.00 0.00 0.00 0.00 0 ksoftirqd/0
04:37:28 PM 0 5 0.00 0.00 0.00 0.00 0 kworker/0:0H
04:37:28 PM 0 7 0.00 0.00 0.00 0.00 0 migration/0
04:37:28 PM 0 8 0.00 0.00 0.00 0.00 0 rcu_bh
04:37:28 PM 0 9 0.00 0.07 0.00 0.07 1 rcu_sched
04:37:28 PM 0 10 0.00 0.00 0.00 0.00 0 lru-add-drain
04:37:28 PM 0 11 0.00 0.00 0.00 0.00 0 watchdog/0
04:37:28 PM 0 12 0.00 0.00 0.00 0.00 1 watchdog/1
04:37:28 PM 0 13 0.00 0.00 0.00 0.00 1 migration/1
04:37:28 PM 0 14 0.00 0.00 0.00 0.00 1 ksoftirqd/1
......
2.5 pidstat -r
-r Report page faults and memory utilization.
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain) 11/24/2022 _x86_64_ (4 CPU)
04:41:20 PM UID PID minflt/s majflt/s VSZ RSS %MEM Command
04:41:20 PM 0 1 0.23 0.00 193952 7080 0.09 systemd
04:41:20 PM 0 356 0.00 0.00 165512 6560 0.08 sshd
04:41:20 PM 0 360 0.00 0.00 165208 5936 0.08 sshd
04:41:20 PM 0 365 0.00 0.00 116784 3432 0.04 bash
04:41:20 PM 0 411 0.00 0.00 72092 2872 0.04 sftp-server
04:41:20 PM 0 482 0.00 0.00 165512 6544 0.08 sshd
04:41:20 PM 0 486 0.00 0.00 165208 5936 0.08 sshd
04:41:20 PM 0 490 0.00 0.00 116916 3660 0.05 bash
04:41:20 PM 0 531 0.00 0.00 72092 2876 0.04 sftp-server
04:41:20 PM 0 2254 0.09 0.00 68504 28784 0.36 systemd-journal
04:41:20 PM 0 2276 0.00 0.00 201084 4180 0.05 lvmetad
04:41:20 PM 0 2297 0.00 0.00 48476 5880 0.07 systemd-udevd
04:41:20 PM 0 4496 0.00 0.00 62044 1292 0.02 auditd
04:41:20 PM 0 4498 0.00 0.00 84552 888 0.01 audispd
04:41:20 PM 0 4499 0.00 0.00 55632 1664 0.02 sedispatch
04:41:20 PM 32 4522 0.00 0.00 73648 1376 0.02 rpcbind
04:41:20 PM 0 4523 0.05 0.00 26460 1840 0.02 systemd-logind
04:41:20 PM 0 4524 0.00 0.00 396624 4604 0.06 accounts-daemon
......
minflt/s
Total number of minor faults the task has made per second, those which have not required loading a memory page from disk.
majflt/s
Total number of major faults the task has made per second, those which have required loading a memory page from disk.
VSZ
Virtual Size: The virtual memory usage of entire task in kilobytes.
RSS
Resident Set Size: The non-swapped physical memory used by the task in kilobytes.
%MEM
The tasks's currently used share of available physical memory.
2.6 pidstat -s
-s Report stack utilization.
显示任务堆栈的使用情况：
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain) 11/24/2022 _x86_64_ (4 CPU)
05:06:00 PM UID PID StkSize StkRef Command
05:06:00 PM 0 1 132 48 systemd
05:06:00 PM 0 356 132 40 sshd
05:06:00 PM 0 360 132 40 sshd
05:06:00 PM 0 365 132 28 bash
05:06:00 PM 0 411 132 32 sftp-server
05:06:00 PM 0 482 132 44 sshd
05:06:00 PM 0 486 132 40 sshd
05:06:00 PM 0 490 132 32 bash
05:06:00 PM 0 531 132 36 sftp-server
05:06:00 PM 0 2254 132 44 systemd-journal
05:06:00 PM 0 2276 132 24 lvmetad
05:06:00 PM 0 2297 132 48 systemd-udevd
05:06:00 PM 0 4496 132 44 auditd
05:06:00 PM 0 4498 132 12 audispd
05:06:00 PM 0 4499 132 20 sedispatch
05:06:00 PM 32 4522 132 20 rpcbind
......
StkSize
The amount of memory in kilobytes reserved for the task as stack, but not necessarily used.
(任务堆栈内存量的总大小，KB 为单位，但不一定使用全部)
StkRef
The amount of memory in kilobytes used as stack, referenced by the task.
(任务引用堆栈的内存量，KB 为单位（任务已经使用的堆栈内存大小）)
2.7 pidstat -w
-w Report task switching activity
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain) 11/24/2022 _x86_64_ (4 CPU)
05:06:29 PM UID PID cswch/s nvcswch/s Command
05:06:29 PM 0 1 0.49 0.01 systemd
05:06:29 PM 0 2 0.00 0.00 kthreadd
05:06:29 PM 0 3 0.27 0.00 ksoftirqd/0
05:06:29 PM 0 5 0.00 0.00 kworker/0:0H
05:06:29 PM 0 7 0.19 0.00 migration/0
05:06:29 PM 0 8 0.00 0.00 rcu_bh
05:06:29 PM 0 9 37.92 0.00 rcu_sched
05:06:29 PM 0 10 0.00 0.00 lru-add-drain
05:06:29 PM 0 11 0.25 0.00 watchdog/0
05:06:29 PM 0 12 0.25 0.00 watchdog/1
05:06:29 PM 0 13 0.18 0.00 migration/1
05:06:29 PM 0 14 0.06 0.00 ksoftirqd/1
05:06:29 PM 0 16 0.00 0.00 kworker/1:0H
05:06:29 PM 0 17 0.25 0.00 watchdog/2
05:06:29 PM 0 18 0.17 0.00 migration/2
05:06:29 PM 0 19 0.16 0.00 ksoftirqd/2
05:06:29 PM 0 21 0.00 0.00 kworker/2:0H
05:06:29 PM 0 22 0.25 0.00 watchdog/3
05:06:29 PM 0 23 0.18 0.00 migration/3
......
cswch/s
Total number of voluntary context switches the task made per second. A voluntary context switch occurs when a task blocks because it requires a resource that is unavail‐
able.
nvcswch/s
Total number of non voluntary context switches the task made per second. A involuntary context switch takes place when a task executes for the duration of its time slice
and then is forced to relinquish the processor.
自愿上下文切换：是指进程无法获取所需资源，即任务处于I/O阻塞态时，导致的上下文切换。比如说， I/O、内存等系统资源不足时，就会发生自愿上下文切换。任务访问该资源，但此时资源没有，处于阻塞态等待资源的到来，在等待期间自愿进行上下文切换。
非自愿上下文切换：则是指进程由于时间片已到等原因，被系统强制调度，进而发生的上下文切换。比如说，大量进程都在争抢 CPU 时，就容易发生非自愿上下文切换。
三、pidstat 数据来源
由于是统计每个任务使用资源的使用情况，其实就是获取 /proc/fd/ 下面的内容：
strace -e trace=open -o pidstat.txt pidstat
......
open("/proc/uptime", O_RDONLY) = 3
open("/proc/stat", O_RDONLY) = 3
open("/proc/1/stat", O_RDONLY) = 4
open("/proc/1/status", O_RDONLY) = 4
open("/proc/2/stat", O_RDONLY) = 4
open("/proc/2/status", O_RDONLY) = 4
open("/proc/3/stat", O_RDONLY) = 4
open("/proc/3/status", O_RDONLY) = 4
open("/proc/5/stat", O_RDONLY) = 4
open("/proc/5/status", O_RDONLY) = 4
open("/proc/7/stat", O_RDONLY) = 4
open("/proc/7/status", O_RDONLY) = 4
open("/proc/8/stat", O_RDONLY) = 4
open("/proc/8/status", O_RDONLY) = 4
open("/proc/9/stat", O_RDONLY) = 4
open("/proc/9/status", O_RDONLY) = 4
open("/proc/10/stat", O_RDONLY) = 4
open("/proc/10/status", O_RDONLY) = 4
open("/proc/11/stat", O_RDONLY) = 4
open("/proc/11/status", O_RDONLY) = 4
open("/proc/12/stat", O_RDONLY) = 4
open("/proc/12/status", O_RDONLY) = 4
......
参考资料
极客时间：Linux 性能优化实战
转载：https://blog.csdn.net/weixin_45030965/article/details/128018075
查看评论