imc.rst 6.3 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199
  1. .. SPDX-License-Identifier: GPL-2.0
  2. .. _imc:
  3. ===================================
  4. IMC (In-Memory Collection Counters)
  5. ===================================
  6. Anju T Sudhakar, 10 May 2019
  7. .. contents::
  8. :depth: 3
  9. Basic overview
  10. ==============
  11. IMC (In-Memory collection counters) is a hardware monitoring facility that
  12. collects large numbers of hardware performance events at Nest level (these are
  13. on-chip but off-core), Core level and Thread level.
  14. The Nest PMU counters are handled by a Nest IMC microcode which runs in the OCC
  15. (On-Chip Controller) complex. The microcode collects the counter data and moves
  16. the nest IMC counter data to memory.
  17. The Core and Thread IMC PMU counters are handled in the core. Core level PMU
  18. counters give us the IMC counters' data per core and thread level PMU counters
  19. give us the IMC counters' data per CPU thread.
  20. OPAL obtains the IMC PMU and supported events information from the IMC Catalog
  21. and passes on to the kernel via the device tree. The event's information
  22. contains:
  23. - Event name
  24. - Event Offset
  25. - Event description
  26. and possibly also:
  27. - Event scale
  28. - Event unit
  29. Some PMUs may have a common scale and unit values for all their supported
  30. events. For those cases, the scale and unit properties for those events must be
  31. inherited from the PMU.
  32. The event offset in the memory is where the counter data gets accumulated.
  33. IMC catalog is available at:
  34. https://github.com/open-power/ima-catalog
  35. The kernel discovers the IMC counters information in the device tree at the
  36. `imc-counters` device node which has a compatible field
  37. `ibm,opal-in-memory-counters`. From the device tree, the kernel parses the PMUs
  38. and their event's information and register the PMU and its attributes in the
  39. kernel.
  40. IMC example usage
  41. =================
  42. .. code-block:: sh
  43. # perf list
  44. [...]
  45. nest_mcs01/PM_MCS01_64B_RD_DISP_PORT01/ [Kernel PMU event]
  46. nest_mcs01/PM_MCS01_64B_RD_DISP_PORT23/ [Kernel PMU event]
  47. [...]
  48. core_imc/CPM_0THRD_NON_IDLE_PCYC/ [Kernel PMU event]
  49. core_imc/CPM_1THRD_NON_IDLE_INST/ [Kernel PMU event]
  50. [...]
  51. thread_imc/CPM_0THRD_NON_IDLE_PCYC/ [Kernel PMU event]
  52. thread_imc/CPM_1THRD_NON_IDLE_INST/ [Kernel PMU event]
  53. To see per chip data for nest_mcs0/PM_MCS_DOWN_128B_DATA_XFER_MC0/:
  54. .. code-block:: sh
  55. # ./perf stat -e "nest_mcs01/PM_MCS01_64B_WR_DISP_PORT01/" -a --per-socket
  56. To see non-idle instructions for core 0:
  57. .. code-block:: sh
  58. # ./perf stat -e "core_imc/CPM_NON_IDLE_INST/" -C 0 -I 1000
  59. To see non-idle instructions for a "make":
  60. .. code-block:: sh
  61. # ./perf stat -e "thread_imc/CPM_NON_IDLE_PCYC/" make
  62. IMC Trace-mode
  63. ===============
  64. POWER9 supports two modes for IMC which are the Accumulation mode and Trace
  65. mode. In Accumulation mode, event counts are accumulated in system Memory.
  66. Hypervisor then reads the posted counts periodically or when requested. In IMC
  67. Trace mode, the 64 bit trace SCOM value is initialized with the event
  68. information. The CPMCxSEL and CPMC_LOAD in the trace SCOM, specifies the event
  69. to be monitored and the sampling duration. On each overflow in the CPMCxSEL,
  70. hardware snapshots the program counter along with event counts and writes into
  71. memory pointed by LDBAR.
  72. LDBAR is a 64 bit special purpose per thread register, it has bits to indicate
  73. whether hardware is configured for accumulation or trace mode.
  74. LDBAR Register Layout
  75. ---------------------
  76. +-------+----------------------+
  77. | 0 | Enable/Disable |
  78. +-------+----------------------+
  79. | 1 | 0: Accumulation Mode |
  80. | +----------------------+
  81. | | 1: Trace Mode |
  82. +-------+----------------------+
  83. | 2:3 | Reserved |
  84. +-------+----------------------+
  85. | 4-6 | PB scope |
  86. +-------+----------------------+
  87. | 7 | Reserved |
  88. +-------+----------------------+
  89. | 8:50 | Counter Address |
  90. +-------+----------------------+
  91. | 51:63 | Reserved |
  92. +-------+----------------------+
  93. TRACE_IMC_SCOM bit representation
  94. ---------------------------------
  95. +-------+------------+
  96. | 0:1 | SAMPSEL |
  97. +-------+------------+
  98. | 2:33 | CPMC_LOAD |
  99. +-------+------------+
  100. | 34:40 | CPMC1SEL |
  101. +-------+------------+
  102. | 41:47 | CPMC2SEL |
  103. +-------+------------+
  104. | 48:50 | BUFFERSIZE |
  105. +-------+------------+
  106. | 51:63 | RESERVED |
  107. +-------+------------+
  108. CPMC_LOAD contains the sampling duration. SAMPSEL and CPMCxSEL determines the
  109. event to count. BUFFERSIZE indicates the memory range. On each overflow,
  110. hardware snapshots the program counter along with event counts and updates the
  111. memory and reloads the CMPC_LOAD value for the next sampling duration. IMC
  112. hardware does not support exceptions, so it quietly wraps around if memory
  113. buffer reaches the end.
  114. *Currently the event monitored for trace-mode is fixed as cycle.*
  115. Trace IMC example usage
  116. =======================
  117. .. code-block:: sh
  118. # perf list
  119. [....]
  120. trace_imc/trace_cycles/ [Kernel PMU event]
  121. To record an application/process with trace-imc event:
  122. .. code-block:: sh
  123. # perf record -e trace_imc/trace_cycles/ yes > /dev/null
  124. [ perf record: Woken up 1 times to write data ]
  125. [ perf record: Captured and wrote 0.012 MB perf.data (21 samples) ]
  126. The `perf.data` generated, can be read using perf report.
  127. Benefits of using IMC trace-mode
  128. ================================
  129. PMI (Performance Monitoring Interrupts) interrupt handling is avoided, since IMC
  130. trace mode snapshots the program counter and updates to the memory. And this
  131. also provide a way for the operating system to do instruction sampling in real
  132. time without PMI processing overhead.
  133. Performance data using `perf top` with and without trace-imc event.
  134. PMI interrupts count when `perf top` command is executed without trace-imc event.
  135. .. code-block:: sh
  136. # grep PMI /proc/interrupts
  137. PMI: 0 0 0 0 Performance monitoring interrupts
  138. # ./perf top
  139. ...
  140. # grep PMI /proc/interrupts
  141. PMI: 39735 8710 17338 17801 Performance monitoring interrupts
  142. # ./perf top -e trace_imc/trace_cycles/
  143. ...
  144. # grep PMI /proc/interrupts
  145. PMI: 39735 8710 17338 17801 Performance monitoring interrupts
  146. That is, the PMI interrupt counts do not increment when using the `trace_imc` event.