Linux Applications Debugging Techniques/Appendices

Tools Overview: http://www.brendangregg.com/Perf/linux_observability_tools.png bpftrace Things to watch for Interrupted calls A number of API functions return an error code if the call was interrupted by a signal. Usually this is not an error by itself and the call should be restarted. For instance: int raccept( int s, struct sockaddr *addr, socklen_t *addrlen ) { int rc; do { rc = accept( s, addr, addrlen ); } while ( rc == -1 && errno == EINTR ); return rc; } The list of interruptible function differs from Unix-like platform to platform. For Linux see signal(7). Note that sem_wait() & co. are on list. Spurious wake-ups Threads waiting on a pthreads condition variable can be waken up even if the condition hs not been met. Upon waking up, the condition should be explicitly checked and return waiting if it is not met. Code review checklist Functional impact Which modules are affected; which functionality is impacted. Which modules & functionality is consumed by the changed code. Which modules/functionality is using it. Side effects. Architecture Scalability Stability Functionality Robustness Technicalities Generic First pass: logic. Correctness. What does it (tries to) do. Side effects. Why is code there? Second pass: error handling & odd conditions. Side effects. Are returned codes checked. Can the code handle them. Arguments: Border/out of bounds/strange values handled. Arg type: convenient? Where is the data coming from? Three levels upstream. Where are the outputs going? Three levels downstream. What are the possible/permitted outputs? Are all handled downstream? Variables: initialized? (over/under)flows: buffers. integral types. Asserted invariants & assumptions. Loops: off-by-one? infinite? Recursion: base case? Third pass: thread safety. Localization? Duplicate information? Math: Small values processed first? NaN checks/std::isnan & friends. Division by zero. C return: resource handling. Use the N functions (ex. s'n'printf vs. sprintf). printf & friends: do formatters match type and number of arguments? Non re-entrant/non thread-safe API Unsafe/race condition afflicted API ( mktemp() ) enums: all cases handled? default returns/logs/throws error Unspecified/undefined/implementation-defined/non-standard behavior or extensions C++ Fourth pass: exception handling: In destructors. Exception catching boundaries? Are resources RAII guarded. Fifth pass: performance. C++11: perfect forwarding & temporaries. How does it scale? Initialized data members. The big 6 (default constructor, destructor, copy/move constructors/operators). Virtual destructor? Resource handling: RAII. Where is created & destructed? Lifetime management? Thorough const & volatile. Are predicates stateless. Use size_t for std::string positions. Why casts? Do not trace under lock Do not destroy objects under lock Platform specific POSIX/SUS Interrupted calls. Spurious wake-ups. Calls known to have race conditions. Tools Source Code Audit CERT C Secure Coding Standard Links Undefined Behavior /proc A pseudo-filesystem exposing information about running processes: # tree /proc/26041 /proc/26041 ... |-- cmdline # Command line |-- cwd -> /current/working/folder/for/PID |-- environ # Program environment variables |-- exe -> /bin/su |-- fd # Open files descriptors | |-- 0 -> /dev/pts/21 | |-- 1 -> /dev/pts/21 | |-- 2 -> /dev/pts/21 | `-- 3 -> socket:[113497835] |-- fdinfo | |-- 0 | |-- 1 | |-- 2 | `-- 3 |-- latency |-- limits |-- maps |-- mem |-- mountinfo |-- mounts |-- mountstats ... # cat /proc/26041/status ... VmPeak: 103276 kB # Max virtual memory reached VmSize: 103196 kB # Current VM VmLck: 0 kB VmHWM: 1492 kB VmRSS: 1488 kB # Live memory used ... Threads: 1 ... References Man page Kernel doc sysstat, sar Site Other tools addr2line Given an address in an executable or an offset in a section of a relocatable object, addr2line translates it into file name and line number. c++filt A tool to demangle symbol names. objdump Disassemble binary, with source code: objdump -C -S -r -R -l <binary> Varia Source Code Audit Links Bunch of tools dstat Abbreviations HLE: Hardware Lock Elision HT: Hyper Threading RTM: Restricted Transactional Memory TSX: Intel Transactional Synchronization Extension (, )