mmiotrace.rst 6.8 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184
  1. ===================================
  2. In-kernel memory-mapped I/O tracing
  3. ===================================
  4. Home page and links to optional user space tools:
  5. https://nouveau.freedesktop.org/wiki/MmioTrace
  6. MMIO tracing was originally developed by Intel around 2003 for their Fault
  7. Injection Test Harness. In Dec 2006 - Jan 2007, using the code from Intel,
  8. Jeff Muizelaar created a tool for tracing MMIO accesses with the Nouveau
  9. project in mind. Since then many people have contributed.
  10. Mmiotrace was built for reverse engineering any memory-mapped IO device with
  11. the Nouveau project as the first real user. Only x86 and x86_64 architectures
  12. are supported.
  13. Out-of-tree mmiotrace was originally modified for mainline inclusion and
  14. ftrace framework by Pekka Paalanen <pq@iki.fi>.
  15. Preparation
  16. -----------
  17. Mmiotrace feature is compiled in by the CONFIG_MMIOTRACE option. Tracing is
  18. disabled by default, so it is safe to have this set to yes. SMP systems are
  19. supported, but tracing is unreliable and may miss events if more than one CPU
  20. is on-line, therefore mmiotrace takes all but one CPU off-line during run-time
  21. activation. You can re-enable CPUs by hand, but you have been warned, there
  22. is no way to automatically detect if you are losing events due to CPUs racing.
  23. Usage Quick Reference
  24. ---------------------
  25. ::
  26. $ mount -t debugfs debugfs /sys/kernel/debug
  27. $ echo mmiotrace > /sys/kernel/debug/tracing/current_tracer
  28. $ cat /sys/kernel/debug/tracing/trace_pipe > mydump.txt &
  29. Start X or whatever.
  30. $ echo "X is up" > /sys/kernel/debug/tracing/trace_marker
  31. $ echo nop > /sys/kernel/debug/tracing/current_tracer
  32. Check for lost events.
  33. Usage
  34. -----
  35. Make sure debugfs is mounted to /sys/kernel/debug.
  36. If not (requires root privileges)::
  37. $ mount -t debugfs debugfs /sys/kernel/debug
  38. Check that the driver you are about to trace is not loaded.
  39. Activate mmiotrace (requires root privileges)::
  40. $ echo mmiotrace > /sys/kernel/debug/tracing/current_tracer
  41. Start storing the trace::
  42. $ cat /sys/kernel/debug/tracing/trace_pipe > mydump.txt &
  43. The 'cat' process should stay running (sleeping) in the background.
  44. Load the driver you want to trace and use it. Mmiotrace will only catch MMIO
  45. accesses to areas that are ioremapped while mmiotrace is active.
  46. During tracing you can place comments (markers) into the trace by
  47. $ echo "X is up" > /sys/kernel/debug/tracing/trace_marker
  48. This makes it easier to see which part of the (huge) trace corresponds to
  49. which action. It is recommended to place descriptive markers about what you
  50. do.
  51. Shut down mmiotrace (requires root privileges)::
  52. $ echo nop > /sys/kernel/debug/tracing/current_tracer
  53. The 'cat' process exits. If it does not, kill it by issuing 'fg' command and
  54. pressing ctrl+c.
  55. Check that mmiotrace did not lose events due to a buffer filling up. Either::
  56. $ grep -i lost mydump.txt
  57. which tells you exactly how many events were lost, or use::
  58. $ dmesg
  59. to view your kernel log and look for "mmiotrace has lost events" warning. If
  60. events were lost, the trace is incomplete. You should enlarge the buffers and
  61. try again. Buffers are enlarged by first seeing how large the current buffers
  62. are::
  63. $ cat /sys/kernel/debug/tracing/buffer_size_kb
  64. gives you a number. Approximately double this number and write it back, for
  65. instance::
  66. $ echo 128000 > /sys/kernel/debug/tracing/buffer_size_kb
  67. Then start again from the top.
  68. If you are doing a trace for a driver project, e.g. Nouveau, you should also
  69. do the following before sending your results::
  70. $ lspci -vvv > lspci.txt
  71. $ dmesg > dmesg.txt
  72. $ tar zcf pciid-nick-mmiotrace.tar.gz mydump.txt lspci.txt dmesg.txt
  73. and then send the .tar.gz file. The trace compresses considerably. Replace
  74. "pciid" and "nick" with the PCI ID or model name of your piece of hardware
  75. under investigation and your nickname.
  76. How Mmiotrace Works
  77. -------------------
  78. Access to hardware IO-memory is gained by mapping addresses from PCI bus by
  79. calling one of the ioremap_*() functions. Mmiotrace is hooked into the
  80. __ioremap() function and gets called whenever a mapping is created. Mapping is
  81. an event that is recorded into the trace log. Note that ISA range mappings
  82. are not caught, since the mapping always exists and is returned directly.
  83. MMIO accesses are recorded via page faults. Just before __ioremap() returns,
  84. the mapped pages are marked as not present. Any access to the pages causes a
  85. fault. The page fault handler calls mmiotrace to handle the fault. Mmiotrace
  86. marks the page present, sets TF flag to achieve single stepping and exits the
  87. fault handler. The instruction that faulted is executed and debug trap is
  88. entered. Here mmiotrace again marks the page as not present. The instruction
  89. is decoded to get the type of operation (read/write), data width and the value
  90. read or written. These are stored to the trace log.
  91. Setting the page present in the page fault handler has a race condition on SMP
  92. machines. During the single stepping other CPUs may run freely on that page
  93. and events can be missed without a notice. Re-enabling other CPUs during
  94. tracing is discouraged.
  95. Trace Log Format
  96. ----------------
  97. The raw log is text and easily filtered with e.g. grep and awk. One record is
  98. one line in the log. A record starts with a keyword, followed by keyword-
  99. dependent arguments. Arguments are separated by a space, or continue until the
  100. end of line. The format for version 20070824 is as follows:
  101. Explanation Keyword Space-separated arguments
  102. ---------------------------------------------------------------------------
  103. read event R width, timestamp, map id, physical, value, PC, PID
  104. write event W width, timestamp, map id, physical, value, PC, PID
  105. ioremap event MAP timestamp, map id, physical, virtual, length, PC, PID
  106. iounmap event UNMAP timestamp, map id, PC, PID
  107. marker MARK timestamp, text
  108. version VERSION the string "20070824"
  109. info for reader LSPCI one line from lspci -v
  110. PCI address map PCIDEV space-separated /proc/bus/pci/devices data
  111. unk. opcode UNKNOWN timestamp, map id, physical, data, PC, PID
  112. Timestamp is in seconds with decimals. Physical is a PCI bus address, virtual
  113. is a kernel virtual address. Width is the data width in bytes and value is the
  114. data value. Map id is an arbitrary id number identifying the mapping that was
  115. used in an operation. PC is the program counter and PID is process id. PC is
  116. zero if it is not recorded. PID is always zero as tracing MMIO accesses
  117. originating in user space memory is not yet supported.
  118. For instance, the following awk filter will pass all 32-bit writes that target
  119. physical addresses in the range [0xfb73ce40, 0xfb800000]
  120. ::
  121. $ awk '/W 4 / { adr=strtonum($5); if (adr >= 0xfb73ce40 &&
  122. adr < 0xfb800000) print; }'
  123. Tools for Developers
  124. --------------------
  125. The user space tools include utilities for:
  126. - replacing numeric addresses and values with hardware register names
  127. - replaying MMIO logs, i.e., re-executing the recorded writes