Wednesday, August 22, 2012

Linux diagnostic utilities

Here's a list of commands and utilities you might find useful to provide various interesting diagnostic information about your Linux system. This is just a starting point, look at the man page on each utility for the full story.
Not all utilities are included by default with most distributions, you may need to install them using your package manager first. For RHEL / CentOS, this often also means adding the EPEL, RPMFusion, and/or RepoForge repositories.
A lot of this is also provided by the gkrellm GUI utility, which is a great real-time monitor for system performance. You might want to enhance gkrellm by configuring it to launch some of the commands below to provide additional detail.

System Resources

uname -a
# The currently running Linux kernel version, architecture, build date
free -mto
# Memory usage in megabytes, free, disk cache, buffers
cat /proc/meminfo
# more memory usage details
echo "2" > /proc/sys/vm/drop_caches
# clear the disk cache (useful before benchmarking your disk read performance)
dstat -a --top-io --top-bio
# Also displays disk and network utilization, along with the name of the most memory and disk-I/O intensive process.

Process Info

# Table of processes, bring up the help ("?") and spend some time playing with the setup... particularly "1", "M/P". Also try making it super colorful with: "A", "Z", "aaa", "Enter", "B". Use at your own peril, but once you have it set up the way you like, use "W" to save that configuration as default.
# Advanced table of processes with historical logging by enabling the atopd service. Access historical logs using `atop -r`, then jump forwards/backwards in time with t/T . Very useful for debugging system and process issues after the fact, and it's very good at highlighting which system resources are constrained (even relatively obscure ones like context switches per sec).
# Identify processes using network I/O
# Identify processes using disk / block device I/O
# What processes are being run by each user
# List of open files by process # Particularly useful for figuring out which process is preventing you from unmounting your USB disk so you can terminate it
pmap -x <PID>
# Show the memory map of a process, including which shared libraries it's using
strace -f <command>
# Run a command and display all system function calls... useful for finding out what a process is doing or trying to do when it isn't responding, what files it's using (filter for the "open" call), etc. Also note you can use it to connect to an existing process with "-p <pid>. Also beware that strace will make the process run slower and perhaps unstable.
ltrace -f <command>
# Like strace, but shows all library function calls.

Hardware Info

# View kernel ring buffer of debug messages, which is mostly about hardware and driver initialization activity. Usually also logged to /var/log/messages
# motherboard / BIOS version information
# PCI bus devices, add "-v" for more detail
# USB devices, add "-v" for more detail
# show device driver kernel modules loaded
cat /proc/interrupts
# show number of interrupts triggered by devices on each CPU
cat /proc/ioports
# show I/O ports reserved by devices
cat /proc/iomem
# show I/O memory ranges reserved by kernel
# environmental sensor data for temperatures, voltages, fan speeds. Might need to install lm-sensors package and perform initial setup using "sensors-detect" script.

Disk & Filesystem Info

# flush all buffered / pending writes to disk
# Show disk free / utilization of each mounted partition
# show which partitions are mounted
cat /proc/mounts
# show which partitions are mounted with options
sfdisk -l
# list all visible disk partitions
smartctl -a /dev/sda
# display hard disk SMART monitoring information, including runtime and number of detected bad blocks ("reallocated sector count")
# list and mark bad blocks on a device. Part of RHEL/CentOS RPMForge sg3_utils package
hdparm -I /dev/sda
# display disk information, including model and serial number. Also check out "sdparm" for SCSI disks
hdparm -Tt /dev/sda
# perform sequential read throughput test
dumpe2fs /dev/sda1
# show ext2/3/4 filesystem information. Use "tune2fs" to change options and features, such as journal size, block size, used/unused block map
du | sort -n
# report disk usage of current directory tree. Also try the GUI utilities such as "filelight"
# How much read/write blocks to each disk device
mdadm --detail
# Software RAID configuration
cat /proc/mdstat
# Software RAID status
blktrace -d /dev/sda -o - | blkparse -i -
# record and display disk head movement (well, sector number) as it reads/writes from a block device. It's also possible to plot this over time using the included script to give a pretty good idea of how many seeks your disk load is generating, as well as how you might optimize it using "readahead" techniques and the like.
Block device benchmarking tool supporting more load patterns.
File system benchmarking tool

Network Info

# display all configured network interfaces
ethtool -i eth0
# show NIC driver and firmware version
ethtool -S eth0
# show NIC statistics
sar -n DEV
# display transmit/receive statistics over time
mii-tool eth0
# show NIC link status
netstat -ap
# state of open network ports, connections, sockets, and associated processes
route -n
# network routing table
iptables -L
# IP filtering tables
tcpdump -i eth0
# sniff network activity on a NIC. Better yet, use the "wireshark" GUI to get lots of useful filtering, reporting, and deep packet inspection
nmap <hostname>
# probe for open ports on a server (clear this with you rIT/security department first!)
ping <hostname>
# measure round trip time to another server over the network
traceroute / tracepath
# discover network path to another server on the network and ping times to all network devices in between
NPtcp # start listener server on remote host NPtcp -h <remotehost>
# netpipe performance benchmark. Adapt the "geplot" script to plot the "np.out" file to view network throughput vs. packet size.

Video Driver Info

less /var/log/Xorg.0.log
# Xorg log file. Also reference config file in /etc/X11/xorg.conf
# X Window System configuration, extensions, and colorspaces
# X Window System OpenGL capabilities
# quick OpenGL test / benchmark
# NVidia driver configuration GUI
# X Rotation and Resolution settings

Audio Driver Info

aplay -l
# list audio hardware devices. It can also play .wav files.
# Volume control settings

Input Device Utilities

# X input events received by a selected window
# X window client properties of a selected window
# virtual X keyboard, can be used to send keyboard events with mouse or touchscreen, or even scripts. Works much better than gok (GNOME Onscreen Keyboard)
# various properties for mouse acceleration, keyboard LED control, sleep timeout, etc. May have to deconflict / disable gnome-screensaver-preferences first if you actually want to use some of these, though.

No comments:

Post a Comment