tlb.rst 3.6 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283
  1. .. SPDX-License-Identifier: GPL-2.0
  2. =======
  3. The TLB
  4. =======
  5. When the kernel unmaps or modified the attributes of a range of
  6. memory, it has two choices:
  7. 1. Flush the entire TLB with a two-instruction sequence. This is
  8. a quick operation, but it causes collateral damage: TLB entries
  9. from areas other than the one we are trying to flush will be
  10. destroyed and must be refilled later, at some cost.
  11. 2. Use the invlpg instruction to invalidate a single page at a
  12. time. This could potentially cost many more instructions, but
  13. it is a much more precise operation, causing no collateral
  14. damage to other TLB entries.
  15. Which method to do depends on a few things:
  16. 1. The size of the flush being performed. A flush of the entire
  17. address space is obviously better performed by flushing the
  18. entire TLB than doing 2^48/PAGE_SIZE individual flushes.
  19. 2. The contents of the TLB. If the TLB is empty, then there will
  20. be no collateral damage caused by doing the global flush, and
  21. all of the individual flush will have ended up being wasted
  22. work.
  23. 3. The size of the TLB. The larger the TLB, the more collateral
  24. damage we do with a full flush. So, the larger the TLB, the
  25. more attractive an individual flush looks. Data and
  26. instructions have separate TLBs, as do different page sizes.
  27. 4. The microarchitecture. The TLB has become a multi-level
  28. cache on modern CPUs, and the global flushes have become more
  29. expensive relative to single-page flushes.
  30. There is obviously no way the kernel can know all these things,
  31. especially the contents of the TLB during a given flush. The
  32. sizes of the flush will vary greatly depending on the workload as
  33. well. There is essentially no "right" point to choose.
  34. You may be doing too many individual invalidations if you see the
  35. invlpg instruction (or instructions _near_ it) show up high in
  36. profiles. If you believe that individual invalidations being
  37. called too often, you can lower the tunable::
  38. /sys/kernel/debug/x86/tlb_single_page_flush_ceiling
  39. This will cause us to do the global flush for more cases.
  40. Lowering it to 0 will disable the use of the individual flushes.
  41. Setting it to 1 is a very conservative setting and it should
  42. never need to be 0 under normal circumstances.
  43. Despite the fact that a single individual flush on x86 is
  44. guaranteed to flush a full 2MB [1]_, hugetlbfs always uses the full
  45. flushes. THP is treated exactly the same as normal memory.
  46. You might see invlpg inside of flush_tlb_mm_range() show up in
  47. profiles, or you can use the trace_tlb_flush() tracepoints. to
  48. determine how long the flush operations are taking.
  49. Essentially, you are balancing the cycles you spend doing invlpg
  50. with the cycles that you spend refilling the TLB later.
  51. You can measure how expensive TLB refills are by using
  52. performance counters and 'perf stat', like this::
  53. perf stat -e
  54. cpu/event=0x8,umask=0x84,name=dtlb_load_misses_walk_duration/,
  55. cpu/event=0x8,umask=0x82,name=dtlb_load_misses_walk_completed/,
  56. cpu/event=0x49,umask=0x4,name=dtlb_store_misses_walk_duration/,
  57. cpu/event=0x49,umask=0x2,name=dtlb_store_misses_walk_completed/,
  58. cpu/event=0x85,umask=0x4,name=itlb_misses_walk_duration/,
  59. cpu/event=0x85,umask=0x2,name=itlb_misses_walk_completed/
  60. That works on an IvyBridge-era CPU (i5-3320M). Different CPUs
  61. may have differently-named counters, but they should at least
  62. be there in some form. You can use pmu-tools 'ocperf list'
  63. (https://github.com/andikleen/pmu-tools) to find the right
  64. counters for a given CPU.
  65. .. [1] A footnote in Intel's SDM "4.10.4.2 Recommended Invalidation"
  66. says: "One execution of INVLPG is sufficient even for a page
  67. with size greater than 4 KBytes."