resctrl_ui.rst 43 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211
  1. .. SPDX-License-Identifier: GPL-2.0
  2. .. include:: <isonum.txt>
  3. ===========================================
  4. User Interface for Resource Control feature
  5. ===========================================
  6. :Copyright: |copy| 2016 Intel Corporation
  7. :Authors: - Fenghua Yu <fenghua.yu@intel.com>
  8. - Tony Luck <tony.luck@intel.com>
  9. - Vikas Shivappa <vikas.shivappa@intel.com>
  10. Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT).
  11. AMD refers to this feature as AMD Platform Quality of Service(AMD QoS).
  12. This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo
  13. flag bits:
  14. ============================================= ================================
  15. RDT (Resource Director Technology) Allocation "rdt_a"
  16. CAT (Cache Allocation Technology) "cat_l3", "cat_l2"
  17. CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2"
  18. CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc"
  19. MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local"
  20. MBA (Memory Bandwidth Allocation) "mba"
  21. ============================================= ================================
  22. To use the feature mount the file system::
  23. # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps]] /sys/fs/resctrl
  24. mount options are:
  25. "cdp":
  26. Enable code/data prioritization in L3 cache allocations.
  27. "cdpl2":
  28. Enable code/data prioritization in L2 cache allocations.
  29. "mba_MBps":
  30. Enable the MBA Software Controller(mba_sc) to specify MBA
  31. bandwidth in MBps
  32. L2 and L3 CDP are controlled separately.
  33. RDT features are orthogonal. A particular system may support only
  34. monitoring, only control, or both monitoring and control. Cache
  35. pseudo-locking is a unique way of using cache control to "pin" or
  36. "lock" data in the cache. Details can be found in
  37. "Cache Pseudo-Locking".
  38. The mount succeeds if either of allocation or monitoring is present, but
  39. only those files and directories supported by the system will be created.
  40. For more details on the behavior of the interface during monitoring
  41. and allocation, see the "Resource alloc and monitor groups" section.
  42. Info directory
  43. ==============
  44. The 'info' directory contains information about the enabled
  45. resources. Each resource has its own subdirectory. The subdirectory
  46. names reflect the resource names.
  47. Each subdirectory contains the following files with respect to
  48. allocation:
  49. Cache resource(L3/L2) subdirectory contains the following files
  50. related to allocation:
  51. "num_closids":
  52. The number of CLOSIDs which are valid for this
  53. resource. The kernel uses the smallest number of
  54. CLOSIDs of all enabled resources as limit.
  55. "cbm_mask":
  56. The bitmask which is valid for this resource.
  57. This mask is equivalent to 100%.
  58. "min_cbm_bits":
  59. The minimum number of consecutive bits which
  60. must be set when writing a mask.
  61. "shareable_bits":
  62. Bitmask of shareable resource with other executing
  63. entities (e.g. I/O). User can use this when
  64. setting up exclusive cache partitions. Note that
  65. some platforms support devices that have their
  66. own settings for cache use which can over-ride
  67. these bits.
  68. "bit_usage":
  69. Annotated capacity bitmasks showing how all
  70. instances of the resource are used. The legend is:
  71. "0":
  72. Corresponding region is unused. When the system's
  73. resources have been allocated and a "0" is found
  74. in "bit_usage" it is a sign that resources are
  75. wasted.
  76. "H":
  77. Corresponding region is used by hardware only
  78. but available for software use. If a resource
  79. has bits set in "shareable_bits" but not all
  80. of these bits appear in the resource groups'
  81. schematas then the bits appearing in
  82. "shareable_bits" but no resource group will
  83. be marked as "H".
  84. "X":
  85. Corresponding region is available for sharing and
  86. used by hardware and software. These are the
  87. bits that appear in "shareable_bits" as
  88. well as a resource group's allocation.
  89. "S":
  90. Corresponding region is used by software
  91. and available for sharing.
  92. "E":
  93. Corresponding region is used exclusively by
  94. one resource group. No sharing allowed.
  95. "P":
  96. Corresponding region is pseudo-locked. No
  97. sharing allowed.
  98. Memory bandwidth(MB) subdirectory contains the following files
  99. with respect to allocation:
  100. "min_bandwidth":
  101. The minimum memory bandwidth percentage which
  102. user can request.
  103. "bandwidth_gran":
  104. The granularity in which the memory bandwidth
  105. percentage is allocated. The allocated
  106. b/w percentage is rounded off to the next
  107. control step available on the hardware. The
  108. available bandwidth control steps are:
  109. min_bandwidth + N * bandwidth_gran.
  110. "delay_linear":
  111. Indicates if the delay scale is linear or
  112. non-linear. This field is purely informational
  113. only.
  114. "thread_throttle_mode":
  115. Indicator on Intel systems of how tasks running on threads
  116. of a physical core are throttled in cases where they
  117. request different memory bandwidth percentages:
  118. "max":
  119. the smallest percentage is applied
  120. to all threads
  121. "per-thread":
  122. bandwidth percentages are directly applied to
  123. the threads running on the core
  124. If RDT monitoring is available there will be an "L3_MON" directory
  125. with the following files:
  126. "num_rmids":
  127. The number of RMIDs available. This is the
  128. upper bound for how many "CTRL_MON" + "MON"
  129. groups can be created.
  130. "mon_features":
  131. Lists the monitoring events if
  132. monitoring is enabled for the resource.
  133. "max_threshold_occupancy":
  134. Read/write file provides the largest value (in
  135. bytes) at which a previously used LLC_occupancy
  136. counter can be considered for re-use.
  137. Finally, in the top level of the "info" directory there is a file
  138. named "last_cmd_status". This is reset with every "command" issued
  139. via the file system (making new directories or writing to any of the
  140. control files). If the command was successful, it will read as "ok".
  141. If the command failed, it will provide more information that can be
  142. conveyed in the error returns from file operations. E.g.
  143. ::
  144. # echo L3:0=f7 > schemata
  145. bash: echo: write error: Invalid argument
  146. # cat info/last_cmd_status
  147. mask f7 has non-consecutive 1-bits
  148. Resource alloc and monitor groups
  149. =================================
  150. Resource groups are represented as directories in the resctrl file
  151. system. The default group is the root directory which, immediately
  152. after mounting, owns all the tasks and cpus in the system and can make
  153. full use of all resources.
  154. On a system with RDT control features additional directories can be
  155. created in the root directory that specify different amounts of each
  156. resource (see "schemata" below). The root and these additional top level
  157. directories are referred to as "CTRL_MON" groups below.
  158. On a system with RDT monitoring the root directory and other top level
  159. directories contain a directory named "mon_groups" in which additional
  160. directories can be created to monitor subsets of tasks in the CTRL_MON
  161. group that is their ancestor. These are called "MON" groups in the rest
  162. of this document.
  163. Removing a directory will move all tasks and cpus owned by the group it
  164. represents to the parent. Removing one of the created CTRL_MON groups
  165. will automatically remove all MON groups below it.
  166. All groups contain the following files:
  167. "tasks":
  168. Reading this file shows the list of all tasks that belong to
  169. this group. Writing a task id to the file will add a task to the
  170. group. If the group is a CTRL_MON group the task is removed from
  171. whichever previous CTRL_MON group owned the task and also from
  172. any MON group that owned the task. If the group is a MON group,
  173. then the task must already belong to the CTRL_MON parent of this
  174. group. The task is removed from any previous MON group.
  175. "cpus":
  176. Reading this file shows a bitmask of the logical CPUs owned by
  177. this group. Writing a mask to this file will add and remove
  178. CPUs to/from this group. As with the tasks file a hierarchy is
  179. maintained where MON groups may only include CPUs owned by the
  180. parent CTRL_MON group.
  181. When the resource group is in pseudo-locked mode this file will
  182. only be readable, reflecting the CPUs associated with the
  183. pseudo-locked region.
  184. "cpus_list":
  185. Just like "cpus", only using ranges of CPUs instead of bitmasks.
  186. When control is enabled all CTRL_MON groups will also contain:
  187. "schemata":
  188. A list of all the resources available to this group.
  189. Each resource has its own line and format - see below for details.
  190. "size":
  191. Mirrors the display of the "schemata" file to display the size in
  192. bytes of each allocation instead of the bits representing the
  193. allocation.
  194. "mode":
  195. The "mode" of the resource group dictates the sharing of its
  196. allocations. A "shareable" resource group allows sharing of its
  197. allocations while an "exclusive" resource group does not. A
  198. cache pseudo-locked region is created by first writing
  199. "pseudo-locksetup" to the "mode" file before writing the cache
  200. pseudo-locked region's schemata to the resource group's "schemata"
  201. file. On successful pseudo-locked region creation the mode will
  202. automatically change to "pseudo-locked".
  203. When monitoring is enabled all MON groups will also contain:
  204. "mon_data":
  205. This contains a set of files organized by L3 domain and by
  206. RDT event. E.g. on a system with two L3 domains there will
  207. be subdirectories "mon_L3_00" and "mon_L3_01". Each of these
  208. directories have one file per event (e.g. "llc_occupancy",
  209. "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
  210. files provide a read out of the current value of the event for
  211. all tasks in the group. In CTRL_MON groups these files provide
  212. the sum for all tasks in the CTRL_MON group and all tasks in
  213. MON groups. Please see example section for more details on usage.
  214. Resource allocation rules
  215. -------------------------
  216. When a task is running the following rules define which resources are
  217. available to it:
  218. 1) If the task is a member of a non-default group, then the schemata
  219. for that group is used.
  220. 2) Else if the task belongs to the default group, but is running on a
  221. CPU that is assigned to some specific group, then the schemata for the
  222. CPU's group is used.
  223. 3) Otherwise the schemata for the default group is used.
  224. Resource monitoring rules
  225. -------------------------
  226. 1) If a task is a member of a MON group, or non-default CTRL_MON group
  227. then RDT events for the task will be reported in that group.
  228. 2) If a task is a member of the default CTRL_MON group, but is running
  229. on a CPU that is assigned to some specific group, then the RDT events
  230. for the task will be reported in that group.
  231. 3) Otherwise RDT events for the task will be reported in the root level
  232. "mon_data" group.
  233. Notes on cache occupancy monitoring and control
  234. ===============================================
  235. When moving a task from one group to another you should remember that
  236. this only affects *new* cache allocations by the task. E.g. you may have
  237. a task in a monitor group showing 3 MB of cache occupancy. If you move
  238. to a new group and immediately check the occupancy of the old and new
  239. groups you will likely see that the old group is still showing 3 MB and
  240. the new group zero. When the task accesses locations still in cache from
  241. before the move, the h/w does not update any counters. On a busy system
  242. you will likely see the occupancy in the old group go down as cache lines
  243. are evicted and re-used while the occupancy in the new group rises as
  244. the task accesses memory and loads into the cache are counted based on
  245. membership in the new group.
  246. The same applies to cache allocation control. Moving a task to a group
  247. with a smaller cache partition will not evict any cache lines. The
  248. process may continue to use them from the old partition.
  249. Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID)
  250. to identify a control group and a monitoring group respectively. Each of
  251. the resource groups are mapped to these IDs based on the kind of group. The
  252. number of CLOSid and RMID are limited by the hardware and hence the creation of
  253. a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID
  254. and creation of "MON" group may fail if we run out of RMIDs.
  255. max_threshold_occupancy - generic concepts
  256. ------------------------------------------
  257. Note that an RMID once freed may not be immediately available for use as
  258. the RMID is still tagged the cache lines of the previous user of RMID.
  259. Hence such RMIDs are placed on limbo list and checked back if the cache
  260. occupancy has gone down. If there is a time when system has a lot of
  261. limbo RMIDs but which are not ready to be used, user may see an -EBUSY
  262. during mkdir.
  263. max_threshold_occupancy is a user configurable value to determine the
  264. occupancy at which an RMID can be freed.
  265. Schemata files - general concepts
  266. ---------------------------------
  267. Each line in the file describes one resource. The line starts with
  268. the name of the resource, followed by specific values to be applied
  269. in each of the instances of that resource on the system.
  270. Cache IDs
  271. ---------
  272. On current generation systems there is one L3 cache per socket and L2
  273. caches are generally just shared by the hyperthreads on a core, but this
  274. isn't an architectural requirement. We could have multiple separate L3
  275. caches on a socket, multiple cores could share an L2 cache. So instead
  276. of using "socket" or "core" to define the set of logical cpus sharing
  277. a resource we use a "Cache ID". At a given cache level this will be a
  278. unique number across the whole system (but it isn't guaranteed to be a
  279. contiguous sequence, there may be gaps). To find the ID for each logical
  280. CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id
  281. Cache Bit Masks (CBM)
  282. ---------------------
  283. For cache resources we describe the portion of the cache that is available
  284. for allocation using a bitmask. The maximum value of the mask is defined
  285. by each cpu model (and may be different for different cache levels). It
  286. is found using CPUID, but is also provided in the "info" directory of
  287. the resctrl file system in "info/{resource}/cbm_mask". Intel hardware
  288. requires that these masks have all the '1' bits in a contiguous block. So
  289. 0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9
  290. and 0xA are not. On a system with a 20-bit mask each bit represents 5%
  291. of the capacity of the cache. You could partition the cache into four
  292. equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
  293. Memory bandwidth Allocation and monitoring
  294. ==========================================
  295. For Memory bandwidth resource, by default the user controls the resource
  296. by indicating the percentage of total memory bandwidth.
  297. The minimum bandwidth percentage value for each cpu model is predefined
  298. and can be looked up through "info/MB/min_bandwidth". The bandwidth
  299. granularity that is allocated is also dependent on the cpu model and can
  300. be looked up at "info/MB/bandwidth_gran". The available bandwidth
  301. control steps are: min_bw + N * bw_gran. Intermediate values are rounded
  302. to the next control step available on the hardware.
  303. The bandwidth throttling is a core specific mechanism on some of Intel
  304. SKUs. Using a high bandwidth and a low bandwidth setting on two threads
  305. sharing a core may result in both threads being throttled to use the
  306. low bandwidth (see "thread_throttle_mode").
  307. The fact that Memory bandwidth allocation(MBA) may be a core
  308. specific mechanism where as memory bandwidth monitoring(MBM) is done at
  309. the package level may lead to confusion when users try to apply control
  310. via the MBA and then monitor the bandwidth to see if the controls are
  311. effective. Below are such scenarios:
  312. 1. User may *not* see increase in actual bandwidth when percentage
  313. values are increased:
  314. This can occur when aggregate L2 external bandwidth is more than L3
  315. external bandwidth. Consider an SKL SKU with 24 cores on a package and
  316. where L2 external is 10GBps (hence aggregate L2 external bandwidth is
  317. 240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20
  318. threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3
  319. bandwidth of 100GBps although the percentage value specified is only 50%
  320. << 100%. Hence increasing the bandwidth percentage will not yield any
  321. more bandwidth. This is because although the L2 external bandwidth still
  322. has capacity, the L3 external bandwidth is fully used. Also note that
  323. this would be dependent on number of cores the benchmark is run on.
  324. 2. Same bandwidth percentage may mean different actual bandwidth
  325. depending on # of threads:
  326. For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4
  327. thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although
  328. they have same percentage bandwidth of 10%. This is simply because as
  329. threads start using more cores in an rdtgroup, the actual bandwidth may
  330. increase or vary although user specified bandwidth percentage is same.
  331. In order to mitigate this and make the interface more user friendly,
  332. resctrl added support for specifying the bandwidth in MBps as well. The
  333. kernel underneath would use a software feedback mechanism or a "Software
  334. Controller(mba_sc)" which reads the actual bandwidth using MBM counters
  335. and adjust the memory bandwidth percentages to ensure::
  336. "actual bandwidth < user specified bandwidth".
  337. By default, the schemata would take the bandwidth percentage values
  338. where as user can switch to the "MBA software controller" mode using
  339. a mount option 'mba_MBps'. The schemata format is specified in the below
  340. sections.
  341. L3 schemata file details (code and data prioritization disabled)
  342. ----------------------------------------------------------------
  343. With CDP disabled the L3 schemata format is::
  344. L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
  345. L3 schemata file details (CDP enabled via mount option to resctrl)
  346. ------------------------------------------------------------------
  347. When CDP is enabled L3 control is split into two separate resources
  348. so you can specify independent masks for code and data like this::
  349. L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
  350. L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
  351. L2 schemata file details
  352. ------------------------
  353. CDP is supported at L2 using the 'cdpl2' mount option. The schemata
  354. format is either::
  355. L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
  356. or
  357. L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
  358. L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
  359. Memory bandwidth Allocation (default mode)
  360. ------------------------------------------
  361. Memory b/w domain is L3 cache.
  362. ::
  363. MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
  364. Memory bandwidth Allocation specified in MBps
  365. ---------------------------------------------
  366. Memory bandwidth domain is L3 cache.
  367. ::
  368. MB:<cache_id0>=bw_MBps0;<cache_id1>=bw_MBps1;...
  369. Reading/writing the schemata file
  370. ---------------------------------
  371. Reading the schemata file will show the state of all resources
  372. on all domains. When writing you only need to specify those values
  373. which you wish to change. E.g.
  374. ::
  375. # cat schemata
  376. L3DATA:0=fffff;1=fffff;2=fffff;3=fffff
  377. L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
  378. # echo "L3DATA:2=3c0;" > schemata
  379. # cat schemata
  380. L3DATA:0=fffff;1=fffff;2=3c0;3=fffff
  381. L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
  382. Cache Pseudo-Locking
  383. ====================
  384. CAT enables a user to specify the amount of cache space that an
  385. application can fill. Cache pseudo-locking builds on the fact that a
  386. CPU can still read and write data pre-allocated outside its current
  387. allocated area on a cache hit. With cache pseudo-locking, data can be
  388. preloaded into a reserved portion of cache that no application can
  389. fill, and from that point on will only serve cache hits. The cache
  390. pseudo-locked memory is made accessible to user space where an
  391. application can map it into its virtual address space and thus have
  392. a region of memory with reduced average read latency.
  393. The creation of a cache pseudo-locked region is triggered by a request
  394. from the user to do so that is accompanied by a schemata of the region
  395. to be pseudo-locked. The cache pseudo-locked region is created as follows:
  396. - Create a CAT allocation CLOSNEW with a CBM matching the schemata
  397. from the user of the cache region that will contain the pseudo-locked
  398. memory. This region must not overlap with any current CAT allocation/CLOS
  399. on the system and no future overlap with this cache region is allowed
  400. while the pseudo-locked region exists.
  401. - Create a contiguous region of memory of the same size as the cache
  402. region.
  403. - Flush the cache, disable hardware prefetchers, disable preemption.
  404. - Make CLOSNEW the active CLOS and touch the allocated memory to load
  405. it into the cache.
  406. - Set the previous CLOS as active.
  407. - At this point the closid CLOSNEW can be released - the cache
  408. pseudo-locked region is protected as long as its CBM does not appear in
  409. any CAT allocation. Even though the cache pseudo-locked region will from
  410. this point on not appear in any CBM of any CLOS an application running with
  411. any CLOS will be able to access the memory in the pseudo-locked region since
  412. the region continues to serve cache hits.
  413. - The contiguous region of memory loaded into the cache is exposed to
  414. user-space as a character device.
  415. Cache pseudo-locking increases the probability that data will remain
  416. in the cache via carefully configuring the CAT feature and controlling
  417. application behavior. There is no guarantee that data is placed in
  418. cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
  419. “locked” data from cache. Power management C-states may shrink or
  420. power off cache. Deeper C-states will automatically be restricted on
  421. pseudo-locked region creation.
  422. It is required that an application using a pseudo-locked region runs
  423. with affinity to the cores (or a subset of the cores) associated
  424. with the cache on which the pseudo-locked region resides. A sanity check
  425. within the code will not allow an application to map pseudo-locked memory
  426. unless it runs with affinity to cores associated with the cache on which the
  427. pseudo-locked region resides. The sanity check is only done during the
  428. initial mmap() handling, there is no enforcement afterwards and the
  429. application self needs to ensure it remains affine to the correct cores.
  430. Pseudo-locking is accomplished in two stages:
  431. 1) During the first stage the system administrator allocates a portion
  432. of cache that should be dedicated to pseudo-locking. At this time an
  433. equivalent portion of memory is allocated, loaded into allocated
  434. cache portion, and exposed as a character device.
  435. 2) During the second stage a user-space application maps (mmap()) the
  436. pseudo-locked memory into its address space.
  437. Cache Pseudo-Locking Interface
  438. ------------------------------
  439. A pseudo-locked region is created using the resctrl interface as follows:
  440. 1) Create a new resource group by creating a new directory in /sys/fs/resctrl.
  441. 2) Change the new resource group's mode to "pseudo-locksetup" by writing
  442. "pseudo-locksetup" to the "mode" file.
  443. 3) Write the schemata of the pseudo-locked region to the "schemata" file. All
  444. bits within the schemata should be "unused" according to the "bit_usage"
  445. file.
  446. On successful pseudo-locked region creation the "mode" file will contain
  447. "pseudo-locked" and a new character device with the same name as the resource
  448. group will exist in /dev/pseudo_lock. This character device can be mmap()'ed
  449. by user space in order to obtain access to the pseudo-locked memory region.
  450. An example of cache pseudo-locked region creation and usage can be found below.
  451. Cache Pseudo-Locking Debugging Interface
  452. ----------------------------------------
  453. The pseudo-locking debugging interface is enabled by default (if
  454. CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl.
  455. There is no explicit way for the kernel to test if a provided memory
  456. location is present in the cache. The pseudo-locking debugging interface uses
  457. the tracing infrastructure to provide two ways to measure cache residency of
  458. the pseudo-locked region:
  459. 1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data
  460. from these measurements are best visualized using a hist trigger (see
  461. example below). In this test the pseudo-locked region is traversed at
  462. a stride of 32 bytes while hardware prefetchers and preemption
  463. are disabled. This also provides a substitute visualization of cache
  464. hits and misses.
  465. 2) Cache hit and miss measurements using model specific precision counters if
  466. available. Depending on the levels of cache on the system the pseudo_lock_l2
  467. and pseudo_lock_l3 tracepoints are available.
  468. When a pseudo-locked region is created a new debugfs directory is created for
  469. it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single
  470. write-only file, pseudo_lock_measure, is present in this directory. The
  471. measurement of the pseudo-locked region depends on the number written to this
  472. debugfs file:
  473. 1:
  474. writing "1" to the pseudo_lock_measure file will trigger the latency
  475. measurement captured in the pseudo_lock_mem_latency tracepoint. See
  476. example below.
  477. 2:
  478. writing "2" to the pseudo_lock_measure file will trigger the L2 cache
  479. residency (cache hits and misses) measurement captured in the
  480. pseudo_lock_l2 tracepoint. See example below.
  481. 3:
  482. writing "3" to the pseudo_lock_measure file will trigger the L3 cache
  483. residency (cache hits and misses) measurement captured in the
  484. pseudo_lock_l3 tracepoint.
  485. All measurements are recorded with the tracing infrastructure. This requires
  486. the relevant tracepoints to be enabled before the measurement is triggered.
  487. Example of latency debugging interface
  488. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  489. In this example a pseudo-locked region named "newlock" was created. Here is
  490. how we can measure the latency in cycles of reading from this region and
  491. visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS
  492. is set::
  493. # :> /sys/kernel/debug/tracing/trace
  494. # echo 'hist:keys=latency' > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/trigger
  495. # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable
  496. # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
  497. # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable
  498. # cat /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/hist
  499. # event histogram
  500. #
  501. # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active]
  502. #
  503. { latency: 456 } hitcount: 1
  504. { latency: 50 } hitcount: 83
  505. { latency: 36 } hitcount: 96
  506. { latency: 44 } hitcount: 174
  507. { latency: 48 } hitcount: 195
  508. { latency: 46 } hitcount: 262
  509. { latency: 42 } hitcount: 693
  510. { latency: 40 } hitcount: 3204
  511. { latency: 38 } hitcount: 3484
  512. Totals:
  513. Hits: 8192
  514. Entries: 9
  515. Dropped: 0
  516. Example of cache hits/misses debugging
  517. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  518. In this example a pseudo-locked region named "newlock" was created on the L2
  519. cache of a platform. Here is how we can obtain details of the cache hits
  520. and misses using the platform's precision counters.
  521. ::
  522. # :> /sys/kernel/debug/tracing/trace
  523. # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable
  524. # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
  525. # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable
  526. # cat /sys/kernel/debug/tracing/trace
  527. # tracer: nop
  528. #
  529. # _-----=> irqs-off
  530. # / _----=> need-resched
  531. # | / _---=> hardirq/softirq
  532. # || / _--=> preempt-depth
  533. # ||| / delay
  534. # TASK-PID CPU# |||| TIMESTAMP FUNCTION
  535. # | | | |||| | |
  536. pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0
  537. Examples for RDT allocation usage
  538. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  539. 1) Example 1
  540. On a two socket machine (one L3 cache per socket) with just four bits
  541. for cache bit masks, minimum b/w of 10% with a memory bandwidth
  542. granularity of 10%.
  543. ::
  544. # mount -t resctrl resctrl /sys/fs/resctrl
  545. # cd /sys/fs/resctrl
  546. # mkdir p0 p1
  547. # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata
  548. # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata
  549. The default resource group is unmodified, so we have access to all parts
  550. of all caches (its schemata file reads "L3:0=f;1=f").
  551. Tasks that are under the control of group "p0" may only allocate from the
  552. "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
  553. Tasks in group "p1" use the "lower" 50% of cache on both sockets.
  554. Similarly, tasks that are under the control of group "p0" may use a
  555. maximum memory b/w of 50% on socket0 and 50% on socket 1.
  556. Tasks in group "p1" may also use 50% memory b/w on both sockets.
  557. Note that unlike cache masks, memory b/w cannot specify whether these
  558. allocations can overlap or not. The allocations specifies the maximum
  559. b/w that the group may be able to use and the system admin can configure
  560. the b/w accordingly.
  561. If resctrl is using the software controller (mba_sc) then user can enter the
  562. max b/w in MB rather than the percentage values.
  563. ::
  564. # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata
  565. # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata
  566. In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w
  567. of 1024MB where as on socket 1 they would use 500MB.
  568. 2) Example 2
  569. Again two sockets, but this time with a more realistic 20-bit mask.
  570. Two real time tasks pid=1234 running on processor 0 and pid=5678 running on
  571. processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy
  572. neighbors, each of the two real-time tasks exclusively occupies one quarter
  573. of L3 cache on socket 0.
  574. ::
  575. # mount -t resctrl resctrl /sys/fs/resctrl
  576. # cd /sys/fs/resctrl
  577. First we reset the schemata for the default group so that the "upper"
  578. 50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by
  579. ordinary tasks::
  580. # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata
  581. Next we make a resource group for our first real time task and give
  582. it access to the "top" 25% of the cache on socket 0.
  583. ::
  584. # mkdir p0
  585. # echo "L3:0=f8000;1=fffff" > p0/schemata
  586. Finally we move our first real time task into this resource group. We
  587. also use taskset(1) to ensure the task always runs on a dedicated CPU
  588. on socket 0. Most uses of resource groups will also constrain which
  589. processors tasks run on.
  590. ::
  591. # echo 1234 > p0/tasks
  592. # taskset -cp 1 1234
  593. Ditto for the second real time task (with the remaining 25% of cache)::
  594. # mkdir p1
  595. # echo "L3:0=7c00;1=fffff" > p1/schemata
  596. # echo 5678 > p1/tasks
  597. # taskset -cp 2 5678
  598. For the same 2 socket system with memory b/w resource and CAT L3 the
  599. schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is
  600. 10):
  601. For our first real time task this would request 20% memory b/w on socket 0.
  602. ::
  603. # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
  604. For our second real time task this would request an other 20% memory b/w
  605. on socket 0.
  606. ::
  607. # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
  608. 3) Example 3
  609. A single socket system which has real-time tasks running on core 4-7 and
  610. non real-time workload assigned to core 0-3. The real-time tasks share text
  611. and data, so a per task association is not required and due to interaction
  612. with the kernel it's desired that the kernel on these cores shares L3 with
  613. the tasks.
  614. ::
  615. # mount -t resctrl resctrl /sys/fs/resctrl
  616. # cd /sys/fs/resctrl
  617. First we reset the schemata for the default group so that the "upper"
  618. 50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0
  619. cannot be used by ordinary tasks::
  620. # echo "L3:0=3ff\nMB:0=50" > schemata
  621. Next we make a resource group for our real time cores and give it access
  622. to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on
  623. socket 0.
  624. ::
  625. # mkdir p0
  626. # echo "L3:0=ffc00\nMB:0=50" > p0/schemata
  627. Finally we move core 4-7 over to the new group and make sure that the
  628. kernel and the tasks running there get 50% of the cache. They should
  629. also get 50% of memory bandwidth assuming that the cores 4-7 are SMT
  630. siblings and only the real time threads are scheduled on the cores 4-7.
  631. ::
  632. # echo F0 > p0/cpus
  633. 4) Example 4
  634. The resource groups in previous examples were all in the default "shareable"
  635. mode allowing sharing of their cache allocations. If one resource group
  636. configures a cache allocation then nothing prevents another resource group
  637. to overlap with that allocation.
  638. In this example a new exclusive resource group will be created on a L2 CAT
  639. system with two L2 cache instances that can be configured with an 8-bit
  640. capacity bitmask. The new exclusive resource group will be configured to use
  641. 25% of each cache instance.
  642. ::
  643. # mount -t resctrl resctrl /sys/fs/resctrl/
  644. # cd /sys/fs/resctrl
  645. First, we observe that the default group is configured to allocate to all L2
  646. cache::
  647. # cat schemata
  648. L2:0=ff;1=ff
  649. We could attempt to create the new resource group at this point, but it will
  650. fail because of the overlap with the schemata of the default group::
  651. # mkdir p0
  652. # echo 'L2:0=0x3;1=0x3' > p0/schemata
  653. # cat p0/mode
  654. shareable
  655. # echo exclusive > p0/mode
  656. -sh: echo: write error: Invalid argument
  657. # cat info/last_cmd_status
  658. schemata overlaps
  659. To ensure that there is no overlap with another resource group the default
  660. resource group's schemata has to change, making it possible for the new
  661. resource group to become exclusive.
  662. ::
  663. # echo 'L2:0=0xfc;1=0xfc' > schemata
  664. # echo exclusive > p0/mode
  665. # grep . p0/*
  666. p0/cpus:0
  667. p0/mode:exclusive
  668. p0/schemata:L2:0=03;1=03
  669. p0/size:L2:0=262144;1=262144
  670. A new resource group will on creation not overlap with an exclusive resource
  671. group::
  672. # mkdir p1
  673. # grep . p1/*
  674. p1/cpus:0
  675. p1/mode:shareable
  676. p1/schemata:L2:0=fc;1=fc
  677. p1/size:L2:0=786432;1=786432
  678. The bit_usage will reflect how the cache is used::
  679. # cat info/L2/bit_usage
  680. 0=SSSSSSEE;1=SSSSSSEE
  681. A resource group cannot be forced to overlap with an exclusive resource group::
  682. # echo 'L2:0=0x1;1=0x1' > p1/schemata
  683. -sh: echo: write error: Invalid argument
  684. # cat info/last_cmd_status
  685. overlaps with exclusive group
  686. Example of Cache Pseudo-Locking
  687. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  688. Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked
  689. region is exposed at /dev/pseudo_lock/newlock that can be provided to
  690. application for argument to mmap().
  691. ::
  692. # mount -t resctrl resctrl /sys/fs/resctrl/
  693. # cd /sys/fs/resctrl
  694. Ensure that there are bits available that can be pseudo-locked, since only
  695. unused bits can be pseudo-locked the bits to be pseudo-locked needs to be
  696. removed from the default resource group's schemata::
  697. # cat info/L2/bit_usage
  698. 0=SSSSSSSS;1=SSSSSSSS
  699. # echo 'L2:1=0xfc' > schemata
  700. # cat info/L2/bit_usage
  701. 0=SSSSSSSS;1=SSSSSS00
  702. Create a new resource group that will be associated with the pseudo-locked
  703. region, indicate that it will be used for a pseudo-locked region, and
  704. configure the requested pseudo-locked region capacity bitmask::
  705. # mkdir newlock
  706. # echo pseudo-locksetup > newlock/mode
  707. # echo 'L2:1=0x3' > newlock/schemata
  708. On success the resource group's mode will change to pseudo-locked, the
  709. bit_usage will reflect the pseudo-locked region, and the character device
  710. exposing the pseudo-locked region will exist::
  711. # cat newlock/mode
  712. pseudo-locked
  713. # cat info/L2/bit_usage
  714. 0=SSSSSSSS;1=SSSSSSPP
  715. # ls -l /dev/pseudo_lock/newlock
  716. crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock
  717. ::
  718. /*
  719. * Example code to access one page of pseudo-locked cache region
  720. * from user space.
  721. */
  722. #define _GNU_SOURCE
  723. #include <fcntl.h>
  724. #include <sched.h>
  725. #include <stdio.h>
  726. #include <stdlib.h>
  727. #include <unistd.h>
  728. #include <sys/mman.h>
  729. /*
  730. * It is required that the application runs with affinity to only
  731. * cores associated with the pseudo-locked region. Here the cpu
  732. * is hardcoded for convenience of example.
  733. */
  734. static int cpuid = 2;
  735. int main(int argc, char *argv[])
  736. {
  737. cpu_set_t cpuset;
  738. long page_size;
  739. void *mapping;
  740. int dev_fd;
  741. int ret;
  742. page_size = sysconf(_SC_PAGESIZE);
  743. CPU_ZERO(&cpuset);
  744. CPU_SET(cpuid, &cpuset);
  745. ret = sched_setaffinity(0, sizeof(cpuset), &cpuset);
  746. if (ret < 0) {
  747. perror("sched_setaffinity");
  748. exit(EXIT_FAILURE);
  749. }
  750. dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR);
  751. if (dev_fd < 0) {
  752. perror("open");
  753. exit(EXIT_FAILURE);
  754. }
  755. mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
  756. dev_fd, 0);
  757. if (mapping == MAP_FAILED) {
  758. perror("mmap");
  759. close(dev_fd);
  760. exit(EXIT_FAILURE);
  761. }
  762. /* Application interacts with pseudo-locked memory @mapping */
  763. ret = munmap(mapping, page_size);
  764. if (ret < 0) {
  765. perror("munmap");
  766. close(dev_fd);
  767. exit(EXIT_FAILURE);
  768. }
  769. close(dev_fd);
  770. exit(EXIT_SUCCESS);
  771. }
  772. Locking between applications
  773. ----------------------------
  774. Certain operations on the resctrl filesystem, composed of read/writes
  775. to/from multiple files, must be atomic.
  776. As an example, the allocation of an exclusive reservation of L3 cache
  777. involves:
  778. 1. Read the cbmmasks from each directory or the per-resource "bit_usage"
  779. 2. Find a contiguous set of bits in the global CBM bitmask that is clear
  780. in any of the directory cbmmasks
  781. 3. Create a new directory
  782. 4. Set the bits found in step 2 to the new directory "schemata" file
  783. If two applications attempt to allocate space concurrently then they can
  784. end up allocating the same bits so the reservations are shared instead of
  785. exclusive.
  786. To coordinate atomic operations on the resctrlfs and to avoid the problem
  787. above, the following locking procedure is recommended:
  788. Locking is based on flock, which is available in libc and also as a shell
  789. script command
  790. Write lock:
  791. A) Take flock(LOCK_EX) on /sys/fs/resctrl
  792. B) Read/write the directory structure.
  793. C) funlock
  794. Read lock:
  795. A) Take flock(LOCK_SH) on /sys/fs/resctrl
  796. B) If success read the directory structure.
  797. C) funlock
  798. Example with bash::
  799. # Atomically read directory structure
  800. $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl
  801. # Read directory contents and create new subdirectory
  802. $ cat create-dir.sh
  803. find /sys/fs/resctrl/ > output.txt
  804. mask = function-of(output.txt)
  805. mkdir /sys/fs/resctrl/newres/
  806. echo mask > /sys/fs/resctrl/newres/schemata
  807. $ flock /sys/fs/resctrl/ ./create-dir.sh
  808. Example with C::
  809. /*
  810. * Example code do take advisory locks
  811. * before accessing resctrl filesystem
  812. */
  813. #include <sys/file.h>
  814. #include <stdlib.h>
  815. void resctrl_take_shared_lock(int fd)
  816. {
  817. int ret;
  818. /* take shared lock on resctrl filesystem */
  819. ret = flock(fd, LOCK_SH);
  820. if (ret) {
  821. perror("flock");
  822. exit(-1);
  823. }
  824. }
  825. void resctrl_take_exclusive_lock(int fd)
  826. {
  827. int ret;
  828. /* release lock on resctrl filesystem */
  829. ret = flock(fd, LOCK_EX);
  830. if (ret) {
  831. perror("flock");
  832. exit(-1);
  833. }
  834. }
  835. void resctrl_release_lock(int fd)
  836. {
  837. int ret;
  838. /* take shared lock on resctrl filesystem */
  839. ret = flock(fd, LOCK_UN);
  840. if (ret) {
  841. perror("flock");
  842. exit(-1);
  843. }
  844. }
  845. void main(void)
  846. {
  847. int fd, ret;
  848. fd = open("/sys/fs/resctrl", O_DIRECTORY);
  849. if (fd == -1) {
  850. perror("open");
  851. exit(-1);
  852. }
  853. resctrl_take_shared_lock(fd);
  854. /* code to read directory contents */
  855. resctrl_release_lock(fd);
  856. resctrl_take_exclusive_lock(fd);
  857. /* code to read and write directory contents */
  858. resctrl_release_lock(fd);
  859. }
  860. Examples for RDT Monitoring along with allocation usage
  861. =======================================================
  862. Reading monitored data
  863. ----------------------
  864. Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would
  865. show the current snapshot of LLC occupancy of the corresponding MON
  866. group or CTRL_MON group.
  867. Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group)
  868. ------------------------------------------------------------------------
  869. On a two socket machine (one L3 cache per socket) with just four bits
  870. for cache bit masks::
  871. # mount -t resctrl resctrl /sys/fs/resctrl
  872. # cd /sys/fs/resctrl
  873. # mkdir p0 p1
  874. # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata
  875. # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata
  876. # echo 5678 > p1/tasks
  877. # echo 5679 > p1/tasks
  878. The default resource group is unmodified, so we have access to all parts
  879. of all caches (its schemata file reads "L3:0=f;1=f").
  880. Tasks that are under the control of group "p0" may only allocate from the
  881. "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
  882. Tasks in group "p1" use the "lower" 50% of cache on both sockets.
  883. Create monitor groups and assign a subset of tasks to each monitor group.
  884. ::
  885. # cd /sys/fs/resctrl/p1/mon_groups
  886. # mkdir m11 m12
  887. # echo 5678 > m11/tasks
  888. # echo 5679 > m12/tasks
  889. fetch data (data shown in bytes)
  890. ::
  891. # cat m11/mon_data/mon_L3_00/llc_occupancy
  892. 16234000
  893. # cat m11/mon_data/mon_L3_01/llc_occupancy
  894. 14789000
  895. # cat m12/mon_data/mon_L3_00/llc_occupancy
  896. 16789000
  897. The parent ctrl_mon group shows the aggregated data.
  898. ::
  899. # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
  900. 31234000
  901. Example 2 (Monitor a task from its creation)
  902. --------------------------------------------
  903. On a two socket machine (one L3 cache per socket)::
  904. # mount -t resctrl resctrl /sys/fs/resctrl
  905. # cd /sys/fs/resctrl
  906. # mkdir p0 p1
  907. An RMID is allocated to the group once its created and hence the <cmd>
  908. below is monitored from its creation.
  909. ::
  910. # echo $$ > /sys/fs/resctrl/p1/tasks
  911. # <cmd>
  912. Fetch the data::
  913. # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
  914. 31789000
  915. Example 3 (Monitor without CAT support or before creating CAT groups)
  916. ---------------------------------------------------------------------
  917. Assume a system like HSW has only CQM and no CAT support. In this case
  918. the resctrl will still mount but cannot create CTRL_MON directories.
  919. But user can create different MON groups within the root group thereby
  920. able to monitor all tasks including kernel threads.
  921. This can also be used to profile jobs cache size footprint before being
  922. able to allocate them to different allocation groups.
  923. ::
  924. # mount -t resctrl resctrl /sys/fs/resctrl
  925. # cd /sys/fs/resctrl
  926. # mkdir mon_groups/m01
  927. # mkdir mon_groups/m02
  928. # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks
  929. # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks
  930. Monitor the groups separately and also get per domain data. From the
  931. below its apparent that the tasks are mostly doing work on
  932. domain(socket) 0.
  933. ::
  934. # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy
  935. 31234000
  936. # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy
  937. 34555
  938. # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy
  939. 31234000
  940. # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy
  941. 32789
  942. Example 4 (Monitor real time tasks)
  943. -----------------------------------
  944. A single socket system which has real time tasks running on cores 4-7
  945. and non real time tasks on other cpus. We want to monitor the cache
  946. occupancy of the real time threads on these cores.
  947. ::
  948. # mount -t resctrl resctrl /sys/fs/resctrl
  949. # cd /sys/fs/resctrl
  950. # mkdir p1
  951. Move the cpus 4-7 over to p1::
  952. # echo f0 > p1/cpus
  953. View the llc occupancy snapshot::
  954. # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy
  955. 11234000