The Linux Kernel/Debugging

⚲ API linux/err.h – helper macros for error pointer handling and propagation linux/errno.h – standard error codes used throughout the kernel. Performance There are many factors that can affect the performance of the Linux kernel, including hardware configurations, software configurations, and workload characteristics. In this context, performance optimization of the Linux kernel involves identifying and addressing performance Bottleneck (software) bottlenecks in the system. This can involve tuning kernel parameters, optimizing system resources, and identifying and fixing bugs and other issues that may be impacting performance. Given the complexity of the Linux kernel and the wide range of factors that can affect performance, performance optimization can be a challenging task. However, with the right tools and techniques, it is possible to significantly improve the performance and reliability of Linux-based systems. Perf_events perf (Linux) Perf_events, short for performance events, is a powerful interface that provides detailed insights into the performance characteristics of software running on a system. By analyzing the data collected by perf_events, developers can identify performance bottlenecks and optimize software to improve performance and reduce resource utilization. Perf_events is designed to be a lightweight, low-overhead monitoring solution that has minimal impact on system performance. 🔧 TODO ⚲ Interfaces 1 perf – performance analysis tools Basic commands: 1 perf-help – display help information about perf 1 perf-top – System profiling tool. 1 perf-record – Run a command and record its profile into perf.data 1 perf-report – Read perf.data (created by perf record) and display the profile Other commands: 1 perf-annotate – Read perf.data (created by perf record) and display annotated code 1 perf-archive – Create archive with object files with build-ids found ... 1 perf-arm-spe – Support for Arm Statistical Profiling Extension within... 1 perf-bench – General framework for benchmark suites 1 perf-buildid-cache – Manage build-id cache. 1 perf-buildid-list – List the buildids in a perf.data file 1 perf-c2c – Shared Data C2C/HITM Analyzer. 1 perf-config – Get and set variables in a configuration file. 1 perf-daemon – Run record sessions on background 1 perf-data – Data file related processing 1 perf-diff – Read perf.data files and display the differential profile 1 perf-dlfilter – Filter sample events using a dynamically loaded shared... 1 perf-evlist – List the event names in a perf.data file 1 perf-ftrace – simple wrapper for kernel's ftrace functionality 1 perf-inject – Filter to augment the events stream with additional in... 1 perf-intel-pt – Support for Intel Processor Trace within perf tools 1 perf-iostat – Show I/O performance metrics 1 perf-kallsyms – Searches running kernel for symbols 1 perf-kmem – Tool to trace/measure kernel memory properties 1 perf-kvm – Tool to trace/measure kvm guest os 1 perf-kwork – Tool to trace/measure kernel work properties (latencies) 1 perf-list – List all symbolic event types 1 perf-lock – Analyze lock events 1 perf-mem – Profile memory accesses 1 perf-probe – Define new dynamic tracepoints 1 perf-sched – Tool to trace/measure scheduler properties (latencies) 1 perf-script – Read perf.data (created by perf record) and display tr... 1 perf-script-perl – Process trace data with a Perl script 1 perf-script-python – Process trace data with a Python script 1 perf-stat – Run a command and gather performance counter statistics 1 perf-test – Runs sanity tests. 1 perf-timechart – Tool to visualize total system behavior during a workload 1 perf-trace – strace inspired tool 1 perf-version – display the version of perf binary ⚙️ Internals 2 perf_event_open – sets up performance monitoring uapi/linux/perf_event.h tools/perf – performance analysis tools linux/perf_event.h kernel/events/core.c – performance events core arch/x86/events – x86 Hardware performance counter PMU drivers (Intel, AMD) kernel/profile.c – simple profiling 📖 References perf (Linux) perf – instruments CPU performance counters, tracepoints, kprobes, and uprobes https://perfwiki.github.io/main/ 📚 Further reading perf Examples Flame Graphs – visualization of profiled stack traces The Unofficial Linux Perf Events Web-Page Getting started with perf, RHEL Flamegraphs Detecting false sharing with perf c2c 🛠️ Utilities Performance Co-Pilot, https://pcp.io/ – Performance Co-Pilot Prometheus (software) Prometheus, https://prometheus.io/ https://github.com/redhat-nfvpe/container-perf-tools https://github.com/brendangregg/perf-tools – performance analysis tools based on Linux perf_events (aka perf) and ftrace readprofile – a tool to read kernel profiling information 📚 Further reading Analyzing system performance with BCC, RHEL stress-ng – exercises various kernel interfaces http://trac.gateworks.com/wiki/linux/profiling Analyzing application performance in RHEL 9 Monitoring and managing system status and performance in RHEL 9 Real-time Linux User space debug interfaces ⚲ Interfaces 1 dmesg – prints or control the kernel ring buffer 2 syslog – system call, which is used to control the kernel printk() buffer 1 strace – system calls and signals tracing tool 2 ptrace – process trace system call 3 klogctl 5 core /sys/kernel/debug/ – debugfs rust/kernel/debugfs.rs – Rust bindings dmesg --console-level <level> gdb /usr/src/linux/vmlinux /proc/kcore /proc/self/stack dynamic admin-guide/dynamic-debug-howto.html debug ⌨️ hands-on: echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control ⚙️ Internals handle_sysrq linux/debugfs.h – debugfs API fs/debugfs – debugfs implementation linux/tracefs.h, fs/tracefs – filesystem backing /sys/kernel/tracing linux/pstore.h, fs/pstore – preserves crash logs across reboots 📚 References Development tools for the kernel dev-tools DebugFS filesystems/debugfs.html, samples/qmi/qmi_sample_client.c Kprobe-based Event Tracing trace/kprobetrace.html Dynamic debug admin-guide/dynamic-debug-howto.html Linux Magic System Request Key Hacks admin-guide/sysrq.html LKMPG: debugfs Magic SysRq key Tracing and logging ⚲ API: User-space interface: 1 dmesg – prints or control the kernel ring buffer 2 syslog – system call, which is used to control the kernel printk() buffer /proc/kmsg https://kernelshark.org/ – front end reader of trace-cmd https://trace-cmd.org/, 1 trace-cmd – CLI for Ftrace trace/ftrace.html – Linux kernel internal tracer /sys/kernel/debug/tracing/ The most commonly used functions linux/printk.h dump_stack – prints the current kernel stack trace for debugging purposes pr_alert – logs an alert-level message, indicating a critical event that requires immediate attention pr_cont – continues printing the current message on the same line pr_crit – logs a critical-level message, indicating a severe condition that might require system halt pr_debug – logs a debug-level message for developers, usually enabled in debug builds pr_devel – logs a developer-specific message, typically used for fine-grained debug purposes, see dynamic admin-guide/dynamic-debug-howto.html pr_emerg – logs an emergency-level message, indicating a serious error that could cause system crash pr_err – logs an error-level message, typically indicating an issue that requires attention pr_err_ratelimited – logs an error-level message with rate limiting to prevent excessive logging pr_fmt – defines a format string for kernel messages pr_info – logs an informational-level message, providing status updates or diagnostics pr_info_ratelimited – logs an informational-level message with rate limiting pr_notice – logs a notice-level message, typically used for events that aren't errors but should be noted pr_warn – logs a warning-level message, indicating a potential issue that doesn't immediately affect system functionality pr_warn_once – logs a warning message once, preventing repeated warnings for the same event pr_warn_ratelimited – logs a warning-level message with rate limiting print_hex_dump – prints a hexdump of data for debugging purposes print_hex_dump_debug – prints a detailed hexdump with debugging-level verbosity printk – the primary function for printing kernel messages with varying severity levels va_format – formats a variable argument list into a string for logging or printing purposes ⌨️ hands-on: echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control; dmesg -w and type on built-in keyboard linux/dev_printk.h – device-specific logging dev_crit – prints a critical-level message for a device dev_dbg – prints a debug-level message for a device if debugging is enabled dev_dbg_ratelimited – prints debug messages for a device with rate limiting dev_err – prints an error-level message for a device dev_err_once – prints an error message for a device only once dev_err_probe – prints an error related to probe failure with standard formatting dev_err_ratelimited – prints error messages for a device with rate limiting dev_fmt – defines a format string used by device-specific printk macros dev_info – prints an informational-level message for a device dev_notice – prints a notice-level message for a device dev_printk – generic function to print kernel messages with specified log level for a device dev_vdbg – prints verbose debug messages for a device if enabled at compile time dev_warn – prints a warning-level message for a device dev_warn_once – prints a warning message for a device only once dev_warn_ratelimited – prints warning messages for a device with rate limiting linux/bug.h, asm-generic/bug.h – BUG, WARN, BUILD_BUG_ON macros WARN_ON WARN ⚙️ Internals printk kernel/printk/printk.c – kernel message logging arch/x86/kernel/traps.c – x86 exception and trap handling lib/dump_stack.c – stack trace dumping linux/kprobes.h, kernel/kprobes.c – dynamic tracing probes trace/kprobes.html kernel/trace – tracing infrastructure scripts/tracing/draw_functrace.py – visualize function call graph from trace kernel logging, kernel tracing samples/ftrace – ftrace usage examples samples/trace_events – trace event examples samples/trace_printk – trace_printk examples linux/instrumentation.h 📚 References: Debugging by printing Message logging with printk core-api/printk-basics.html Dynamic debug admin-guide/dynamic-debug-howto.html SystemTap 1 stap – systemtap script translator/driver strace 1 strace – trace system calls and signals LTTng ftrace Linux Tracing Technologies trace Tracepoint Analysis trace/tracepoint-analysis.html Function Tracer trace/ftrace.html – function, latency and event tracing Event Tracing trace/events.html Using ftrace to hook to functions trace/ftrace-uses.html Fprobe - Function entry/exit probe trace/fprobe.html Kprobes trace/kprobes.html Kprobe-based Event Tracing trace/kprobetrace.html Uprobe-tracer: Uprobe-based Event Tracing trace/uprobetracer.html Using the Linux Kernel Tracepoints trace/tracepoints.html Subsystem Trace Points: kmem trace/events-kmem.html Subsystem Trace Points: power trace/events-power.html NMI Trace Events trace/events-nmi.html In-kernel memory-mapped I/O tracing trace/mmiotrace.html Event Histograms trace/histogram.html Histogram Design Notes trace/histogram-design.html Boot-time tracing trace/boottime-trace.html Hardware Latency Detector trace/hwlat_detector.html Intel(R) Trace Hub (TH) trace/intel_th.html Lockless Ring Buffer Design trace/ring-buffer-design.html System Trace Module trace/stm.html CoreSight - ARM Hardware Trace trace/coresight 🔧 TODO. 🚀 advanced features linux/kmemleak.h – memory leak detector pr_cont - continues a previous log message in the same line print_hex_dump_bytes print_hex_dump_debug dump_stack CONFIG_PRINTK_CALLER CONFIG_DEBUG_KERNEL CONFIG_DEBUG_INFO https://git.kernel.org/pub/scm/libs/libtrace/ 📚 Further reading https://lore.kernel.org/linux-trace-kernel/ https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git/ Memory debugging ⚙️ Internals: mm/kasan – KernelAddressSANitizer dev-tools/kasan.html, detects out-of-bounds and use-after-free bugs mm/kfence – Kernel Electric-Fence dev-tools/kfence.html, low-overhead sampling-based memory safety detector linux/kmemleak.h – memory leak detector dev-tools/kmemleak.html, mm/kmemleak.c linux/kcsan-checks.h – KCSAN - Kernel Concurrency Sanitizer dev-tools/kcsan.html, kernel/kcsan kernel/gcov – gcov code coverage dev-tools/gcov.html kgdb and kdb ⚲ Interfaces linux/kgdb.h linux/kdb.h ⚙️ Internals kernel/debug – kgdb/kdb debugger 📚 References Using kgdb, kdb and the kernel debugger internals dev-tools/kgdb.html kdump (Linux) kdump – kernel crash dump mechanism kdump admin-guide/kdump /proc/sys/kernel/panic – seconds before auto-reboot on panic (0 = no reboot) /proc/sys/kernel/panic_on_oops – panic on kernel oops /sys/kernel/kexec_crash_loaded – whether a crash kernel is loaded /sys/kernel/kexec_crash_size – reserved memory for crash kernel kdumpctl status, kdumpctl estimate – check kdump readiness (RHEL/Fedora) 8 crash – Analyze Linux crash dump data or a live system eBPF ⚲ API: 2 bpf ↪ kernel/bpf/syscall.c linux/bpf.h – BPF core definitions bpftool – inspect and manage eBPF programs and maps bpftool prog list – loaded programs bpftool map list – active maps bpftool net list – networking attachments bpftool cgroup tree – cgroup BPF program attachments sos ebpf plugin – diagnostics collection for eBPF ⚙️ Internals: kernel/bpf – BPF subsystem core arch/x86/net – x86 BPF JIT compiler net/bpf – BPF networking hooks 📖 References eBPF and BPF bpf 📚 Further reading 7 bpf-helpers Linux Extended BPF (eBPF) Tracing Tools bpftrace – High-level tracing language for Linux eBPF BCC – Tools for BPF-based Linux IO analysis, networking, monitoring, and more Example of trace.py 8 stapbpf eBPF Programming for Linux Kernel Tracing lockdep - Runtime locking correctness validator locking/lockdep-design.html Watchdogs ../Softdog Driver dev_watchdog – network device watchdog The NMI watchdog lockup detectors: ⚲ API /proc/sys/kernel/nmi_watchdog /proc/sys/kernel/soft_watchdog /proc/sys/kernel/watchdog /proc/sys/kernel/watchdog_cpumask /proc/sys/kernel/watchdog_thresh /proc/sys/kernel/hardlockup_all_cpu_backtrace /proc/sys/kernel/hardlockup_panic /proc/sys/kernel/softlockup_all_cpu_backtrace /proc/sys/kernel/softlockup_panic linux/nmi.h 👁️ Example lib/test_lockup.c – test module to generate lockups Provoke NMI watchdog without panic: echo 0 > /proc/sys/kernel/hardlockup_panic insmod test_lockup.ko disable_irq=1 time_secs=13 ⚙️ Internals kernel/watchdog.c – detects hard and soft lockups on a system kernel/watchdog_perf.c – detects hard lockups on a system using perf kernel/watchdog_buddy.c – coordinates watchdog checks across CPUs linux/hung_task.h, kernel/hung_task.c – detects tasks stuck in uninterruptible sleep 📚 References Documentation for /proc/sys/kernel/ admin-guide/sysctl/kernel.html Softlockup detector and hardlockup detector (aka nmi_watchdog) admin-guide/lockup-watchdogs.html kernel parameters: nmi_watchdog nmi_watchdog nowatchdog nowatchdog%09 nosoftlockup nosoftlockup softlockup_panic softlockup_panic ... ⚙️ Internals arch/x86/kernel/traps.c – x86 exception and trap handling linux/panic.h, kernel/panic.c – kernel panic, oops and BUG handling linux/coredump.h, fs/coredump.c – 5 core dump generation Kernel health counters: /proc/sys/kernel/tainted – bitmask of kernel taint flags (proprietary modules, forced loads, machine checks, etc.) /sys/kernel/oops_count – number of kernel oopses since boot /sys/kernel/warn_count – number of kernel warnings since boot collected by sos kernel plugin for diagnostics 📖 References for debugging Ramoops oops/panic logger admin-guide/ramoops.html pstore block oops/panic logger admin-guide/pstore-blk.html Fault injection fault-injection Bisecting a bug admin-guide/bug-bisect.html Development tools for the kernel dev-tools Kernel Testing Guide dev-tools/testing-overview.html Checkpatch dev-tools/checkpatch.html, scripts/checkpatch.pl Selftests dev-tools/kselftest.html, tools/testing/selftests linux/tracepoint.h Tracking down and identifying problems admin-guide/index.html#tracking-down-and-identifying-problems Bug hunting admin-guide/bug-hunting.html ... 📚 Further reading Linux kernel oops kdump (Linux) https://deepwiki.com/torvalds/linux/2.4-kernel-tracing-and-profiling https://drgn.readthedocs.io/ – programmable debugger https://crash-utility.github.io/ https://wiki.ubuntu.com/Kernel/Debugging Intel VTune Profiler Linux Applications Debugging Techniques