Language

Colour scheme

Cheat sheet

Linux debugging

The commands you need when something is broken. Written for AlmaLinux / RHEL 9, with the Debian / Ubuntu equivalents one click away.

System overviewSystem overview

  • hostnamectl

    Hostname, OS, kernel, virtualisation

  • cat /etc/os-release

    Distribution and exact version

  • uptime

    Uptime and load average (1/5/15 min)

  • who -b; last reboot | head

    Last boot time and reboot history

  • free -h

    Memory and swap usage

  • lscpu

    CPU model, cores, architecture

  • ps aux --sort=-%cpu | head

    Top CPU consumers

  • ps aux --sort=-%mem | head

    Top memory consumers

  • top

    Live processes (press 1 for per-core, M sort by memory, P by CPU)

  • dnf install epel-release && dnf install htop

    Install htop (RHEL: from EPEL)

  • apt install htop

    Install htop

  • vmstat 1 5

    CPU, memory, swap and I/O wait, 5 samples

  • dnf install sysstat && iostat -xz 1

    Per-disk utilisation and latency

  • apt install sysstat && iostat -xz 1

    Per-disk utilisation and latency

  • dmesg -T --level=err,warn

    Kernel errors and warnings with timestamps

  • dmesg -T | grep -i -E 'oom|killed process'

    Processes killed by the OOM killer

Services (systemd)Services (systemd)

  • systemctl status <service>

    State, main PID and the last log lines

  • systemctl --failed

    All failed units

  • systemctl restart <service>

    Restart a service

  • systemctl enable --now <service>

    Enable: start now and on every boot

  • systemctl disable --now <service>

    Disable: stop now and no longer start at boot

  • systemctl is-enabled <service>

    Starts at boot? (enabled / disabled / masked)

  • systemctl mask --now <service>

    careful Block completely: cannot be started, not even as a dependency

  • systemctl unmask <service>

    Undo mask (then enable again if needed)

  • systemctl list-units --type=service --state=running

    Running services

  • systemctl cat <service>

    Unit file including drop-in overrides

  • systemctl edit <service>

    Create a drop-in override (survives package updates)

  • systemctl daemon-reload

    Reload unit files after editing them

  • systemctl list-timers

    Scheduled timers and their next run

  • systemd-analyze blame | head

    What slowed down the last boot

LogsLogs

  • journalctl -u <service> -f

    Follow the log of one service

  • journalctl -u <service> --since "1 hour ago"

    Service log of the last hour

  • journalctl -p err -b

    All errors since the current boot

  • journalctl -b -1 -p warning

    Warnings of the previous boot (e.g. after a crash)

  • journalctl -k -b

    Kernel messages of this boot

  • journalctl --disk-usage

    Space used by the journal

  • journalctl --vacuum-time=14d

    careful Delete journal entries older than 14 days

  • tail -f /var/log/messages

    General system log

  • tail -f /var/log/syslog

    General system log (Debian 12: install rsyslog, otherwise use journalctl)

  • tail -f /var/log/secure

    SSH logins, sudo, authentication

  • tail -f /var/log/auth.log

    SSH logins, sudo, authentication

  • grep 'Failed password' /var/log/secure | tail

    Failed SSH logins

  • grep 'Failed password' /var/log/auth.log | tail

    Failed SSH logins

  • tail -50 /var/log/dnf.log

    Recent package installs and updates

  • tail -50 /var/log/apt/history.log

    Recent package installs and updates

  • tail -f /var/log/audit/audit.log

    Audit log (SELinux denials, logins, sudo)

NetworkNetwork

  • ip -br a

    Interfaces and IP addresses, one line each

  • ip r

    Routing table and default gateway

  • ip -s link show <interface>

    Packet, error and drop counters

  • nmcli device status

    Interfaces managed by NetworkManager

  • cat /etc/network/interfaces

    Interface config (Debian server: ifupdown; Ubuntu: netplan get)

  • nmcli connection show <connection>

    Full connection settings (IP, DNS, gateway)

  • networkctl status

    Link state if systemd-networkd is used (Ubuntu)

  • nmcli connection up <connection>

    careful Re-apply a connection after changes

  • systemctl restart networking

    careful Re-apply /etc/network/interfaces (Ubuntu: netplan apply)

  • cat /etc/resolv.conf

    DNS servers in use

  • ss -tulpn

    Listening TCP/UDP ports and their process

  • ss -tnp state established

    Established TCP connections

  • ping -c 4 <host>

    Basic reachability

  • tracepath <host>

    Path to a host (installed by default, no root needed)

  • dnf install mtr && mtr <host>

    Live traceroute with packet loss per hop

  • apt install mtr-tiny && mtr <host>

    Live traceroute with packet loss per hop

  • dnf install bind-utils && dig <domain>

    DNS lookup (add @1.1.1.1 to ask a specific server)

  • apt install dnsutils && dig <domain>

    DNS lookup (add @1.1.1.1 to ask a specific server)

  • curl -sv -o /dev/null https://<host>

    HTTP(S) request with TLS handshake and headers

  • dnf install nmap-ncat && nc -zv <host> <port>

    Is a TCP port reachable?

  • apt install netcat-openbsd && nc -zv <host> <port>

    Is a TCP port reachable?

  • timeout 3 bash -c "</dev/tcp/<host>/<port>" && echo open

    Port test without any extra tools

  • tcpdump -i any -nn port <port>

    Capture traffic on a port (install tcpdump)

  • ethtool <interface>

    Link speed, duplex, link detected

Firewall (firewalld)Firewall (ufw / nftables)

  • firewall-cmd --state

    Is the firewall running?

  • ufw status verbose

    ufw state and rules (apt install ufw, not installed by default)

  • systemctl enable --now firewalld

    Enable the firewall (now and at boot)

  • ufw allow OpenSSH && ufw enable

    Enable the firewall. Allow SSH first or you lock yourself out

  • systemctl disable --now firewalld

    careful Disable the firewall (for testing only)

  • ufw disable

    careful Disable the firewall (for testing only)

  • firewall-cmd --list-all

    Rules of the default zone

  • ufw status numbered

    Rules with numbers (for deleting)

  • firewall-cmd --get-active-zones

    Zones and their interfaces

  • firewall-cmd --permanent --add-service=https && firewall-cmd --reload

    Allow a service permanently

  • ufw allow 443/tcp

    Allow a service permanently

  • firewall-cmd --permanent --remove-service=https && firewall-cmd --reload

    careful Remove an allowed service

  • ufw delete allow 443/tcp

    careful Remove an allowed service

  • firewall-cmd --permanent --add-port=<port>/tcp && firewall-cmd --reload

    Allow a port permanently

  • ufw allow <port>/tcp

    Allow a port permanently

  • firewall-cmd --permanent --remove-port=<port>/tcp && firewall-cmd --reload

    careful Remove an allowed port

  • ufw delete allow <port>/tcp

    careful Remove an allowed port

  • firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=<ip> port port=22 protocol=tcp accept' && firewall-cmd --reload

    Allow a port only from one source IP

  • ufw allow from <ip> to any port 22 proto tcp

    Allow a port only from one source IP

  • firewall-cmd --permanent --remove-rich-rule='rule family=ipv4 source address=<ip> port port=22 protocol=tcp accept' && firewall-cmd --reload

    careful Remove that source-IP rule (same text as when adding)

  • ufw delete allow from <ip> to any port 22 proto tcp

    careful Remove that source-IP rule (same text as when adding)

  • ufw delete <rule-number>

    careful Remove a rule by its number (see ufw status numbered)

  • nft list ruleset

    The actual kernel ruleset (nftables) behind the tools

SELinuxAppArmor

  • getenforce

    Enforcing, Permissive or Disabled

  • aa-status

    Loaded profiles and their mode

  • ausearch -m avc -ts recent

    Recent SELinux denials

  • journalctl -k | grep -i 'apparmor="DENIED"'

    Recent AppArmor denials

  • dnf install setroubleshoot-server && sealert -a /var/log/audit/audit.log

    Denials explained, with the suggested fix

  • ls -Z <path>

    SELinux context of files

  • restorecon -Rv <path>

    Reset file contexts, fixes most "moved files" problems

  • semanage fcontext -a -t httpd_sys_content_t '<path>(/.*)?' && restorecon -Rv <path>

    Permanently label a custom web directory

  • semanage fcontext -d '<path>(/.*)?' && restorecon -Rv <path>

    Remove that custom label again

  • semanage port -l | grep http

    Ports a service type may bind to

  • semanage port -a -t http_port_t -p tcp <port>

    Allow a service on a non-standard port

  • semanage port -d -t http_port_t -p tcp <port>

    Remove that port permission again

  • getsebool -a | grep httpd

    SELinux booleans for a service

  • setsebool -P httpd_can_network_connect on

    Enable a boolean: let web servers/proxies make outbound connections

  • setsebool -P httpd_can_network_connect off

    Disable that boolean again

  • setenforce 0

    careful Permissive until reboot. For testing only, then fix the cause

  • setenforce 1

    Back to enforcing

  • aa-complain /etc/apparmor.d/<profile>

    careful Profile to complain mode (log only), for testing

  • aa-enforce /etc/apparmor.d/<profile>

    Profile back to enforce mode

Packages (dnf / rpm)Packages (apt / dpkg)

  • dnf check-update

    Available updates

  • apt update && apt list --upgradable

    Available updates

  • dnf upgrade

    Install all updates

  • apt update && apt upgrade

    Install all updates

  • dnf upgrade --security

    Security updates only

  • unattended-upgrade --dry-run -d

    What unattended-upgrades would install

  • dnf install <package>

    Install a package

  • apt install <package>

    Install a package

  • dnf provides '*/bin/<command>'

    Which package contains a command?

  • apt install apt-file && apt-file update && apt-file search bin/<command>

    Which package contains a command?

  • rpm -qf <path>

    Which installed package owns a file?

  • dpkg -S <path>

    Which installed package owns a file?

  • rpm -qa | grep <name>

    Is a package installed, which version?

  • dpkg -l | grep <name>

    Is a package installed, which version?

  • rpm -V <package>

    Files changed since installation

  • dpkg --verify <package>

    Files changed since installation

  • dnf history

    Package transaction history

  • zgrep -h Commandline /var/log/apt/history.log*

    Package transaction history

  • dnf history undo <id>

    careful Roll back a transaction

  • dnf repolist --all

    Repositories and whether they are enabled

  • apt-cache policy

    Enabled repositories

  • dnf config-manager --set-enabled <repo>

    Enable a repository (e.g. crb, needed by many EPEL packages)

  • dnf config-manager --set-disabled <repo>

    Disable a repository

  • dnf install dnf-utils && needs-restarting -r

    Is a reboot required after updates?

  • cat /var/run/reboot-required 2>/dev/null || echo "no reboot needed"

    Is a reboot required after updates?

  • needs-restarting -s

    Services that need a restart after updates

  • apt install needrestart && needrestart -r l

    Services that need a restart after updates

Disks & filesystemsDisks & filesystems

  • df -hT

    Free space per filesystem, with type

  • df -i

    Free inodes: "disk full" with space left?

  • du -xh --max-depth=1 <path> 2>/dev/null | sort -h

    Biggest directories (stays on one filesystem)

  • find / -xdev -type f -size +1G -exec ls -lh {} + 2>/dev/null

    Files larger than 1 GB

  • lsof +L1

    Deleted files still held open (space not freed until restart)

  • lsblk -f

    Disks, partitions, filesystems, mount points

  • findmnt --verify

    Check /etc/fstab before rebooting

  • pvs; vgs; lvs

    LVM physical volumes, volume groups, logical volumes

  • growpart /dev/<disk> <partition-number>

    Grow a partition after enlarging the virtual disk (cloud-utils-growpart)

  • growpart /dev/<disk> <partition-number>

    Grow a partition after enlarging the virtual disk (cloud-guest-utils)

  • lvextend -r -L +<size>G /dev/<vg>/<lv>

    Grow a logical volume and its filesystem in one step

  • xfs_info /

    Filesystem details (RHEL default: XFS)

  • tune2fs -l /dev/<partition>

    Filesystem details (Debian default: ext4)

  • smartctl -a /dev/<disk>

    Disk health (install smartmontools)

Users & accessUsers & access

  • id <user>

    UID, groups

  • sudo -l -U <user>

    What a user may run with sudo

  • usermod -aG wheel <user>

    Grant sudo rights (group wheel)

  • usermod -aG sudo <user>

    Grant sudo rights (group sudo)

  • gpasswd -d <user> wheel

    Revoke sudo rights (takes effect at next login)

  • gpasswd -d <user> sudo

    Revoke sudo rights (takes effect at next login)

  • last -n 20

    Recent logins

  • lastb -n 20

    Recent failed logins (root)

  • chage -l <user>

    Password age and expiry

  • faillock --user <user>

    Locked out after failed logins?

  • faillock --user <user> --reset

    Unlock the account

  • passwd -S <user>

    Password status (locked, set, expired)

  • sshd -t

    Test sshd_config before restarting sshd

  • ssh -vvv <user>@<host>

    Debug a failing SSH login from the client side

Time & certificatesTime & certificates

  • timedatectl

    Time, time zone, NTP sync status

  • chronyc tracking

    Clock offset and sync quality (chrony)

  • timedatectl timesync-status

    Sync status (systemd-timesyncd, Debian default)

  • chronyc sources -v

    NTP servers in use

  • timedatectl show-timesync --all

    NTP servers in use

  • timedatectl set-timezone Europe/Vienna

    Set the time zone

  • openssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates

    Certificate of a remote server

  • openssl x509 -in <cert.pem> -noout -text

    Inspect a certificate file

  • cp <ca.crt> /etc/pki/ca-trust/source/anchors/ && update-ca-trust

    Trust an internal CA system-wide

  • cp <ca.crt> /usr/local/share/ca-certificates/ && update-ca-certificates

    Trust an internal CA system-wide

Same in the terminal

$ curl tools.otw.solutions/linux/selinux

Whole sheet in the terminal: curl tools.otw.solutions/linux | less · Debian: add ?distro=debian