taskstats.rst 8.0 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180
  1. =============================
  2. Per-task statistics interface
  3. =============================
  4. Taskstats is a netlink-based interface for sending per-task and
  5. per-process statistics from the kernel to userspace.
  6. Taskstats was designed for the following benefits:
  7. - efficiently provide statistics during lifetime of a task and on its exit
  8. - unified interface for multiple accounting subsystems
  9. - extensibility for use by future accounting patches
  10. Terminology
  11. -----------
  12. "pid", "tid" and "task" are used interchangeably and refer to the standard
  13. Linux task defined by struct task_struct. per-pid stats are the same as
  14. per-task stats.
  15. "tgid", "process" and "thread group" are used interchangeably and refer to the
  16. tasks that share an mm_struct i.e. the traditional Unix process. Despite the
  17. use of tgid, there is no special treatment for the task that is thread group
  18. leader - a process is deemed alive as long as it has any task belonging to it.
  19. Usage
  20. -----
  21. To get statistics during a task's lifetime, userspace opens a unicast netlink
  22. socket (NETLINK_GENERIC family) and sends commands specifying a pid or a tgid.
  23. The response contains statistics for a task (if pid is specified) or the sum of
  24. statistics for all tasks of the process (if tgid is specified).
  25. To obtain statistics for tasks which are exiting, the userspace listener
  26. sends a register command and specifies a cpumask. Whenever a task exits on
  27. one of the cpus in the cpumask, its per-pid statistics are sent to the
  28. registered listener. Using cpumasks allows the data received by one listener
  29. to be limited and assists in flow control over the netlink interface and is
  30. explained in more detail below.
  31. If the exiting task is the last thread exiting its thread group,
  32. an additional record containing the per-tgid stats is also sent to userspace.
  33. The latter contains the sum of per-pid stats for all threads in the thread
  34. group, both past and present.
  35. getdelays.c is a simple utility demonstrating usage of the taskstats interface
  36. for reporting delay accounting statistics. Users can register cpumasks,
  37. send commands and process responses, listen for per-tid/tgid exit data,
  38. write the data received to a file and do basic flow control by increasing
  39. receive buffer sizes.
  40. Interface
  41. ---------
  42. The user-kernel interface is encapsulated in include/linux/taskstats.h
  43. To avoid this documentation becoming obsolete as the interface evolves, only
  44. an outline of the current version is given. taskstats.h always overrides the
  45. description here.
  46. struct taskstats is the common accounting structure for both per-pid and
  47. per-tgid data. It is versioned and can be extended by each accounting subsystem
  48. that is added to the kernel. The fields and their semantics are defined in the
  49. taskstats.h file.
  50. The data exchanged between user and kernel space is a netlink message belonging
  51. to the NETLINK_GENERIC family and using the netlink attributes interface.
  52. The messages are in the format::
  53. +----------+- - -+-------------+-------------------+
  54. | nlmsghdr | Pad | genlmsghdr | taskstats payload |
  55. +----------+- - -+-------------+-------------------+
  56. The taskstats payload is one of the following three kinds:
  57. 1. Commands: Sent from user to kernel. Commands to get data on
  58. a pid/tgid consist of one attribute, of type TASKSTATS_CMD_ATTR_PID/TGID,
  59. containing a u32 pid or tgid in the attribute payload. The pid/tgid denotes
  60. the task/process for which userspace wants statistics.
  61. Commands to register/deregister interest in exit data from a set of cpus
  62. consist of one attribute, of type
  63. TASKSTATS_CMD_ATTR_REGISTER/DEREGISTER_CPUMASK and contain a cpumask in the
  64. attribute payload. The cpumask is specified as an ascii string of
  65. comma-separated cpu ranges e.g. to listen to exit data from cpus 1,2,3,5,7,8
  66. the cpumask would be "1-3,5,7-8". If userspace forgets to deregister interest
  67. in cpus before closing the listening socket, the kernel cleans up its interest
  68. set over time. However, for the sake of efficiency, an explicit deregistration
  69. is advisable.
  70. 2. Response for a command: sent from the kernel in response to a userspace
  71. command. The payload is a series of three attributes of type:
  72. a) TASKSTATS_TYPE_AGGR_PID/TGID : attribute containing no payload but indicates
  73. a pid/tgid will be followed by some stats.
  74. b) TASKSTATS_TYPE_PID/TGID: attribute whose payload is the pid/tgid whose stats
  75. are being returned.
  76. c) TASKSTATS_TYPE_STATS: attribute with a struct taskstats as payload. The
  77. same structure is used for both per-pid and per-tgid stats.
  78. 3. New message sent by kernel whenever a task exits. The payload consists of a
  79. series of attributes of the following type:
  80. a) TASKSTATS_TYPE_AGGR_PID: indicates next two attributes will be pid+stats
  81. b) TASKSTATS_TYPE_PID: contains exiting task's pid
  82. c) TASKSTATS_TYPE_STATS: contains the exiting task's per-pid stats
  83. d) TASKSTATS_TYPE_AGGR_TGID: indicates next two attributes will be tgid+stats
  84. e) TASKSTATS_TYPE_TGID: contains tgid of process to which task belongs
  85. f) TASKSTATS_TYPE_STATS: contains the per-tgid stats for exiting task's process
  86. per-tgid stats
  87. --------------
  88. Taskstats provides per-process stats, in addition to per-task stats, since
  89. resource management is often done at a process granularity and aggregating task
  90. stats in userspace alone is inefficient and potentially inaccurate (due to lack
  91. of atomicity).
  92. However, maintaining per-process, in addition to per-task stats, within the
  93. kernel has space and time overheads. To address this, the taskstats code
  94. accumulates each exiting task's statistics into a process-wide data structure.
  95. When the last task of a process exits, the process level data accumulated also
  96. gets sent to userspace (along with the per-task data).
  97. When a user queries to get per-tgid data, the sum of all other live threads in
  98. the group is added up and added to the accumulated total for previously exited
  99. threads of the same thread group.
  100. Extending taskstats
  101. -------------------
  102. There are two ways to extend the taskstats interface to export more
  103. per-task/process stats as patches to collect them get added to the kernel
  104. in future:
  105. 1. Adding more fields to the end of the existing struct taskstats. Backward
  106. compatibility is ensured by the version number within the
  107. structure. Userspace will use only the fields of the struct that correspond
  108. to the version its using.
  109. 2. Defining separate statistic structs and using the netlink attributes
  110. interface to return them. Since userspace processes each netlink attribute
  111. independently, it can always ignore attributes whose type it does not
  112. understand (because it is using an older version of the interface).
  113. Choosing between 1. and 2. is a matter of trading off flexibility and
  114. overhead. If only a few fields need to be added, then 1. is the preferable
  115. path since the kernel and userspace don't need to incur the overhead of
  116. processing new netlink attributes. But if the new fields expand the existing
  117. struct too much, requiring disparate userspace accounting utilities to
  118. unnecessarily receive large structures whose fields are of no interest, then
  119. extending the attributes structure would be worthwhile.
  120. Flow control for taskstats
  121. --------------------------
  122. When the rate of task exits becomes large, a listener may not be able to keep
  123. up with the kernel's rate of sending per-tid/tgid exit data leading to data
  124. loss. This possibility gets compounded when the taskstats structure gets
  125. extended and the number of cpus grows large.
  126. To avoid losing statistics, userspace should do one or more of the following:
  127. - increase the receive buffer sizes for the netlink sockets opened by
  128. listeners to receive exit data.
  129. - create more listeners and reduce the number of cpus being listened to by
  130. each listener. In the extreme case, there could be one listener for each cpu.
  131. Users may also consider setting the cpu affinity of the listener to the subset
  132. of cpus to which it listens, especially if they are listening to just one cpu.
  133. Despite these measures, if the userspace receives ENOBUFS error messages
  134. indicated overflow of receive buffers, it should take measures to handle the
  135. loss of data.