The Linux Kernel/Multitasking/Real-time

RT preemption The Linux Foundation's Real-Time Linux (RTL) collaborative project is focused on improving the real-time capabilities of Linux and advancing the adoption of real-time Linux in various industries, including aerospace, automotive, robotics, and telecommunications. Parameter CONFIG_PREEMPT_RT enables real-time preemption. RT scheduling policies Scheduling policies for RT: SCHED_FIFO, SCHED_RR implemented in kernel/sched/rt.c SCHED_DEADLINE implemented in kernel/sched/deadline.c API: 1 chrt – manipulate the real-time attributes of a process 2 sched_rr_get_interval – get the SCHED_RR interval for the named process 2 sched_setscheduler, sched_getscheduler – set and get scheduling policy/parameters 2 sched_get_priority_min, sched_get_priority_max – get static priority range RT scheduler tunables: /proc/sys/kernel/sched_rt_period_us – period over which RT task bandwidth is measured (default 1000000 µs = 1 s) /proc/sys/kernel/sched_rt_runtime_us – maximum time RT tasks may consume per period (default 950000 µs); set to -1 to disable the RT bandwidth limit /sys/kernel/realtime – indicates the running kernel has PREEMPT_RT enabled (present only on RT kernels) /sys/devices/system/clocksource/clocksource0/current_clocksource – active clocksource, critical for RT timing accuracy 🛠️ Utilities tuna – view and change thread attributes (affinity, scheduling policy, priority) RHEL tuna guide sos kernelrt plugin – diagnostics collection for RT kernel Priority inversion occurs when a high-priority task blocks on a resource held by a low-priority task, while a medium-priority task preempts the low-priority one indefinitely. The kernel mitigates this with rt_mutex which implements Priority inheritance – temporarily boosting the lock holder's priority to that of the highest-priority waiter. RT synchronization ⚲ APIs migrate_disable + spin_lock local_lock calls migrate_disable(); rt_spin_lock(this_cpu_ptr((__lock))); 📖 References Usage of migrate_disable PREEMPT_RT caveats: spinlock_t, rwlock_t, migrate_disable and local_lock locking/locktypes.html#spinlock-t-and-rwlock-t ⚙️ Internals Spinlock linux/spinlock_rt.h used via linux/spinlock_types.h spinlock_t rt_mutex_base rt_spin_lock ... __rt_spin_lock ... rtlock_lock ... kernel/locking/spinlock_rt.c – RT-aware spinlock implementation RT Mutex linux/rtmutex.h used via linux/mutex_types.h rt_mutex_base rt_mutex rt_mutex_lock ... kernel/locking/rtmutex_api.c – RT mutex kernel API kernel/locking/rtmutex_common.h – RT mutex internal definitions kernel/locking/rtmutex.c – RT mutex core with priority inheritance rwbase_rt linux/rwbase_rt.h used via linux/rwlock_types.h rwlock_t rwbase_rt – used to implement real-time read/write locks rt_read_trylock ... kernel/locking/rwbase_rt.c – RT rw_semaphore/rwlock base kernel/locking/spinlock_rt.c – RT-aware spinlock implementation Testing RT capabilities The testing process for Real-Time Linux typically involves several key aspects. First and foremost, it is crucial to verify the accuracy and stability of the system's timekeeping mechanisms. Precise time management is fundamental to real-time applications, and any inaccuracies can lead to timing errors and compromise the system's real-time capabilities. Another essential aspect of testing is evaluating the system's scheduling algorithms. Real-Time Linux employs advanced scheduling policies to prioritize critical tasks and ensure their timely execution. Testing the scheduler involves assessing its ability to allocate resources efficiently, handle task prioritization correctly, and prevent resource contention or priority inversion scenarios. Furthermore, latency measurement is a critical part of Real-Time Linux testing. Latency refers to the time delay between the occurrence of an event and the system's response to it. In real-time applications, minimizing latency is crucial to achieving timely and predictable behavior. Testing latency involves measuring the time it takes for the system to respond to various stimuli and identifying any sources of delay or unpredictability. Additionally, stress testing plays a significant role in assessing the system's robustness under heavy workloads. It involves subjecting the Real-Time Linux system to high levels of concurrent activities, intense computational loads, and input/output operations to evaluate its performance, responsiveness, and stability. Stress testing helps identify potential bottlenecks, resource limitations, or issues that might degrade the real-time behavior of the system. RTLA RTLA – The realtime Linux analysis tool: rtla timerlat tools/rtla/rtla-timerlat.html – CLI for the kernel's timerlat tracer trace/timerlat-tracer.html rtla osnoise tools/rtla/rtla-osnoise.html – CLI for the kernel's osnoise tracer trace/osnoise-tracer.html. Kernel function run_osnoise measures time with function trace_clock_local in loop. rtla hwnoise tools/rtla/rtla-hwnoise.html – CLI for the osnoise tracer trace/osnoise-tracer.html with interrupts disabled Implementation: tools/tracing/rtla and kernel/trace/trace_osnoise.c Linux scheduling latency debug and analysis RT-Tests RT-Tests, source, @gitlab cyclictest some RT-Tests man pages: cyclictest – measures 2 clock_nanosleep or 2 nanosleep delay hwlatdetect – CLI for /sys/kernel/tracing/hwlat_detector trace/hwlat_detector.html / kernel/trace/trace_hwlat.c. Kernel function kthread_fn measures time delays with function trace_clock_local in loop. oslat – measures delay with Time_Stamp_Counter RDTSC in busy loop hackbench – scheduler benchmark/stress test ftrace Testing latencies with the ftrace - Function Tracer. tracing_max_latency the irqsoff trace/ftrace.html#irqsoff, preemptoff trace/ftrace.html#preemptoff, preemptirqsoff trace/ftrace.html#preemptirqsoff tracers tools/tracing/latency – latency measurement tools CONFIG_IRQSOFF_TRACER – interrupts-off latency tracer CONFIG_PREEMPT_TRACER – preemption-off latency tracer CONFIG_SCHED_TRACER – scheduling latency tracer Other tests 1 perf-intel-pt – Branch trace Intel Processor Trace for low-overhead instruction-level tracing, tools/perf/Documentation/perf-intel-pt.txt Using rtla with perf processor trace RT Tracing Tools with eBPF realtime https://www.latencytop.org/, kernel/latencytop.c rteval – RT workload and latency evaluation in a container RT optimizations CONFIG_PSI_DEFAULT_DISABLED – disable PSI accounting/psi.html Check: ls /proc/pressure/ should fail ... 📚 Further reading: https://lore.kernel.org/linux-rt-users/ https://lore.kernel.org/linux-trace-kernel/ 📖 References Real-time preemption core-api/real-time/index.html Documentation for /proc/sys/kernel/ admin-guide/sysctl/kernel.html – sched_rt_period_us, sched_rt_runtime_us https://realtime-linux.org/ Getting Started with PREEMPT_RT Guide A Checklist for Real-Time Applications in Linux the Real-Time Linux wiki CPU partitioning and isolation linux-stable-rt.git linux-rt-devel.git 📚 Further reading about real-time Linux: News from PREEMPT_RT – LPC 2025, post-mainline developments https://deepwiki.com/torvalds/linux/2.1-process-scheduler Introduction to Real-Time Linux: Unleashing Deterministic Computing Power Management and Scheduling in the Linux Kernel (OSPM) Realtime@LWN Realtime kernel patchset, Arch Linux https://www.kernel.org/pub/linux/kernel/projects/rt/ - RT patches for upstream kernel High Precision Event Timer (HPET) Demystifying the Real-Time Linux Scheduling Latency RHEL for RT Understanding RHEL for RT – affinity, SCHED_DEADLINE cpusets, RT socket options, timerlat, runtime monitors Optimizing for low latency: SCHED_DEADLINE parameters BIOS tuning – disable power management, error correction, SMI Runtime verification – monitors and reactors CPU isolation – tuned profiles, isolated_cores, nohz_full RCU callback offloading Scheduling problems – throttling, starvation Linux subsystems related to real-time Linux kernel#Scheduling and preemption Linux kernel scheduling and preemption Interrupts Deferred works Non-maskable interrupt handler (NMI) System management interrupt (SMI) 7 sched latency @ LKML PREEMPT_RT @ LKML QA about PREEMPT_RT, LPC'23, State of the onion, pdf 💾 Historical The PREEMPT_RT patch has been fully merged into the mainline Linux kernel, starting from version 6.12.