topology.rst 7.2 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234
  1. .. SPDX-License-Identifier: GPL-2.0
  2. ============
  3. x86 Topology
  4. ============
  5. This documents and clarifies the main aspects of x86 topology modelling and
  6. representation in the kernel. Update/change when doing changes to the
  7. respective code.
  8. The architecture-agnostic topology definitions are in
  9. Documentation/admin-guide/cputopology.rst. This file holds x86-specific
  10. differences/specialities which must not necessarily apply to the generic
  11. definitions. Thus, the way to read up on Linux topology on x86 is to start
  12. with the generic one and look at this one in parallel for the x86 specifics.
  13. Needless to say, code should use the generic functions - this file is *only*
  14. here to *document* the inner workings of x86 topology.
  15. Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.
  16. The main aim of the topology facilities is to present adequate interfaces to
  17. code which needs to know/query/use the structure of the running system wrt
  18. threads, cores, packages, etc.
  19. The kernel does not care about the concept of physical sockets because a
  20. socket has no relevance to software. It's an electromechanical component. In
  21. the past a socket always contained a single package (see below), but with the
  22. advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
  23. there might be still references to sockets in the code, but they are of
  24. historical nature and should be cleaned up.
  25. The topology of a system is described in the units of:
  26. - packages
  27. - cores
  28. - threads
  29. Package
  30. =======
  31. Packages contain a number of cores plus shared resources, e.g. DRAM
  32. controller, shared caches etc.
  33. Modern systems may also use the term 'Die' for package.
  34. AMD nomenclature for package is 'Node'.
  35. Package-related topology information in the kernel:
  36. - cpuinfo_x86.x86_max_cores:
  37. The number of cores in a package. This information is retrieved via CPUID.
  38. - cpuinfo_x86.x86_max_dies:
  39. The number of dies in a package. This information is retrieved via CPUID.
  40. - cpuinfo_x86.cpu_die_id:
  41. The physical ID of the die. This information is retrieved via CPUID.
  42. - cpuinfo_x86.phys_proc_id:
  43. The physical ID of the package. This information is retrieved via CPUID
  44. and deduced from the APIC IDs of the cores in the package.
  45. Modern systems use this value for the socket. There may be multiple
  46. packages within a socket. This value may differ from cpu_die_id.
  47. - cpuinfo_x86.logical_proc_id:
  48. The logical ID of the package. As we do not trust BIOSes to enumerate the
  49. packages in a consistent way, we introduced the concept of logical package
  50. ID so we can sanely calculate the number of maximum possible packages in
  51. the system and have the packages enumerated linearly.
  52. - topology_max_packages():
  53. The maximum possible number of packages in the system. Helpful for per
  54. package facilities to preallocate per package information.
  55. - cpu_llc_id:
  56. A per-CPU variable containing:
  57. - On Intel, the first APIC ID of the list of CPUs sharing the Last Level
  58. Cache
  59. - On AMD, the Node ID or Core Complex ID containing the Last Level
  60. Cache. In general, it is a number identifying an LLC uniquely on the
  61. system.
  62. Cores
  63. =====
  64. A core consists of 1 or more threads. It does not matter whether the threads
  65. are SMT- or CMT-type threads.
  66. AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
  67. "core".
  68. Core-related topology information in the kernel:
  69. - smp_num_siblings:
  70. The number of threads in a core. The number of threads in a package can be
  71. calculated by::
  72. threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings
  73. Threads
  74. =======
  75. A thread is a single scheduling unit. It's the equivalent to a logical Linux
  76. CPU.
  77. AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
  78. uses "thread".
  79. Thread-related topology information in the kernel:
  80. - topology_core_cpumask():
  81. The cpumask contains all online threads in the package to which a thread
  82. belongs.
  83. The number of online threads is also printed in /proc/cpuinfo "siblings."
  84. - topology_sibling_cpumask():
  85. The cpumask contains all online threads in the core to which a thread
  86. belongs.
  87. - topology_logical_package_id():
  88. The logical package ID to which a thread belongs.
  89. - topology_physical_package_id():
  90. The physical package ID to which a thread belongs.
  91. - topology_core_id();
  92. The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
  93. "core_id."
  94. System topology examples
  95. ========================
  96. .. note::
  97. The alternative Linux CPU enumeration depends on how the BIOS enumerates the
  98. threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
  99. That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
  100. the same whether threads are enabled or not. That's merely an implementation
  101. detail and has no practical impact.
  102. 1) Single Package, Single Core::
  103. [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
  104. 2) Single Package, Dual Core
  105. a) One thread per core::
  106. [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
  107. -> [core 1] -> [thread 0] -> Linux CPU 1
  108. b) Two threads per core::
  109. [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
  110. -> [thread 1] -> Linux CPU 1
  111. -> [core 1] -> [thread 0] -> Linux CPU 2
  112. -> [thread 1] -> Linux CPU 3
  113. Alternative enumeration::
  114. [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
  115. -> [thread 1] -> Linux CPU 2
  116. -> [core 1] -> [thread 0] -> Linux CPU 1
  117. -> [thread 1] -> Linux CPU 3
  118. AMD nomenclature for CMT systems::
  119. [node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
  120. -> [Compute Unit Core 1] -> Linux CPU 1
  121. -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
  122. -> [Compute Unit Core 1] -> Linux CPU 3
  123. 4) Dual Package, Dual Core
  124. a) One thread per core::
  125. [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
  126. -> [core 1] -> [thread 0] -> Linux CPU 1
  127. [package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
  128. -> [core 1] -> [thread 0] -> Linux CPU 3
  129. b) Two threads per core::
  130. [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
  131. -> [thread 1] -> Linux CPU 1
  132. -> [core 1] -> [thread 0] -> Linux CPU 2
  133. -> [thread 1] -> Linux CPU 3
  134. [package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
  135. -> [thread 1] -> Linux CPU 5
  136. -> [core 1] -> [thread 0] -> Linux CPU 6
  137. -> [thread 1] -> Linux CPU 7
  138. Alternative enumeration::
  139. [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
  140. -> [thread 1] -> Linux CPU 4
  141. -> [core 1] -> [thread 0] -> Linux CPU 1
  142. -> [thread 1] -> Linux CPU 5
  143. [package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
  144. -> [thread 1] -> Linux CPU 6
  145. -> [core 1] -> [thread 0] -> Linux CPU 3
  146. -> [thread 1] -> Linux CPU 7
  147. AMD nomenclature for CMT systems::
  148. [node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
  149. -> [Compute Unit Core 1] -> Linux CPU 1
  150. -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
  151. -> [Compute Unit Core 1] -> Linux CPU 3
  152. [node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
  153. -> [Compute Unit Core 1] -> Linux CPU 5
  154. -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
  155. -> [Compute Unit Core 1] -> Linux CPU 7