events.rst 40 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073
  1. =============
  2. Event Tracing
  3. =============
  4. :Author: Theodore Ts'o
  5. :Updated: Li Zefan and Tom Zanussi
  6. 1. Introduction
  7. ===============
  8. Tracepoints (see Documentation/trace/tracepoints.rst) can be used
  9. without creating custom kernel modules to register probe functions
  10. using the event tracing infrastructure.
  11. Not all tracepoints can be traced using the event tracing system;
  12. the kernel developer must provide code snippets which define how the
  13. tracing information is saved into the tracing buffer, and how the
  14. tracing information should be printed.
  15. 2. Using Event Tracing
  16. ======================
  17. 2.1 Via the 'set_event' interface
  18. ---------------------------------
  19. The events which are available for tracing can be found in the file
  20. /sys/kernel/debug/tracing/available_events.
  21. To enable a particular event, such as 'sched_wakeup', simply echo it
  22. to /sys/kernel/debug/tracing/set_event. For example::
  23. # echo sched_wakeup >> /sys/kernel/debug/tracing/set_event
  24. .. Note:: '>>' is necessary, otherwise it will firstly disable all the events.
  25. To disable an event, echo the event name to the set_event file prefixed
  26. with an exclamation point::
  27. # echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event
  28. To disable all events, echo an empty line to the set_event file::
  29. # echo > /sys/kernel/debug/tracing/set_event
  30. To enable all events, echo ``*:*`` or ``*:`` to the set_event file::
  31. # echo *:* > /sys/kernel/debug/tracing/set_event
  32. The events are organized into subsystems, such as ext4, irq, sched,
  33. etc., and a full event name looks like this: <subsystem>:<event>. The
  34. subsystem name is optional, but it is displayed in the available_events
  35. file. All of the events in a subsystem can be specified via the syntax
  36. ``<subsystem>:*``; for example, to enable all irq events, you can use the
  37. command::
  38. # echo 'irq:*' > /sys/kernel/debug/tracing/set_event
  39. 2.2 Via the 'enable' toggle
  40. ---------------------------
  41. The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchy
  42. of directories.
  43. To enable event 'sched_wakeup'::
  44. # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
  45. To disable it::
  46. # echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
  47. To enable all events in sched subsystem::
  48. # echo 1 > /sys/kernel/debug/tracing/events/sched/enable
  49. To enable all events::
  50. # echo 1 > /sys/kernel/debug/tracing/events/enable
  51. When reading one of these enable files, there are four results:
  52. - 0 - all events this file affects are disabled
  53. - 1 - all events this file affects are enabled
  54. - X - there is a mixture of events enabled and disabled
  55. - ? - this file does not affect any event
  56. 2.3 Boot option
  57. ---------------
  58. In order to facilitate early boot debugging, use boot option::
  59. trace_event=[event-list]
  60. event-list is a comma separated list of events. See section 2.1 for event
  61. format.
  62. 3. Defining an event-enabled tracepoint
  63. =======================================
  64. See The example provided in samples/trace_events
  65. 4. Event formats
  66. ================
  67. Each trace event has a 'format' file associated with it that contains
  68. a description of each field in a logged event. This information can
  69. be used to parse the binary trace stream, and is also the place to
  70. find the field names that can be used in event filters (see section 5).
  71. It also displays the format string that will be used to print the
  72. event in text mode, along with the event name and ID used for
  73. profiling.
  74. Every event has a set of ``common`` fields associated with it; these are
  75. the fields prefixed with ``common_``. The other fields vary between
  76. events and correspond to the fields defined in the TRACE_EVENT
  77. definition for that event.
  78. Each field in the format has the form::
  79. field:field-type field-name; offset:N; size:N;
  80. where offset is the offset of the field in the trace record and size
  81. is the size of the data item, in bytes.
  82. For example, here's the information displayed for the 'sched_wakeup'
  83. event::
  84. # cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format
  85. name: sched_wakeup
  86. ID: 60
  87. format:
  88. field:unsigned short common_type; offset:0; size:2;
  89. field:unsigned char common_flags; offset:2; size:1;
  90. field:unsigned char common_preempt_count; offset:3; size:1;
  91. field:int common_pid; offset:4; size:4;
  92. field:int common_tgid; offset:8; size:4;
  93. field:char comm[TASK_COMM_LEN]; offset:12; size:16;
  94. field:pid_t pid; offset:28; size:4;
  95. field:int prio; offset:32; size:4;
  96. field:int success; offset:36; size:4;
  97. field:int cpu; offset:40; size:4;
  98. print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,
  99. REC->prio, REC->success, REC->cpu
  100. This event contains 10 fields, the first 5 common and the remaining 5
  101. event-specific. All the fields for this event are numeric, except for
  102. 'comm' which is a string, a distinction important for event filtering.
  103. 5. Event filtering
  104. ==================
  105. Trace events can be filtered in the kernel by associating boolean
  106. 'filter expressions' with them. As soon as an event is logged into
  107. the trace buffer, its fields are checked against the filter expression
  108. associated with that event type. An event with field values that
  109. 'match' the filter will appear in the trace output, and an event whose
  110. values don't match will be discarded. An event with no filter
  111. associated with it matches everything, and is the default when no
  112. filter has been set for an event.
  113. 5.1 Expression syntax
  114. ---------------------
  115. A filter expression consists of one or more 'predicates' that can be
  116. combined using the logical operators '&&' and '||'. A predicate is
  117. simply a clause that compares the value of a field contained within a
  118. logged event with a constant value and returns either 0 or 1 depending
  119. on whether the field value matched (1) or didn't match (0)::
  120. field-name relational-operator value
  121. Parentheses can be used to provide arbitrary logical groupings and
  122. double-quotes can be used to prevent the shell from interpreting
  123. operators as shell metacharacters.
  124. The field-names available for use in filters can be found in the
  125. 'format' files for trace events (see section 4).
  126. The relational-operators depend on the type of the field being tested:
  127. The operators available for numeric fields are:
  128. ==, !=, <, <=, >, >=, &
  129. And for string fields they are:
  130. ==, !=, ~
  131. The glob (~) accepts a wild card character (\*,?) and character classes
  132. ([). For example::
  133. prev_comm ~ "*sh"
  134. prev_comm ~ "sh*"
  135. prev_comm ~ "*sh*"
  136. prev_comm ~ "ba*sh"
  137. If the field is a pointer that points into user space (for example
  138. "filename" from sys_enter_openat), then you have to append ".ustring" to the
  139. field name::
  140. filename.ustring ~ "password"
  141. As the kernel will have to know how to retrieve the memory that the pointer
  142. is at from user space.
  143. 5.2 Setting filters
  144. -------------------
  145. A filter for an individual event is set by writing a filter expression
  146. to the 'filter' file for the given event.
  147. For example::
  148. # cd /sys/kernel/debug/tracing/events/sched/sched_wakeup
  149. # echo "common_preempt_count > 4" > filter
  150. A slightly more involved example::
  151. # cd /sys/kernel/debug/tracing/events/signal/signal_generate
  152. # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter
  153. If there is an error in the expression, you'll get an 'Invalid
  154. argument' error when setting it, and the erroneous string along with
  155. an error message can be seen by looking at the filter e.g.::
  156. # cd /sys/kernel/debug/tracing/events/signal/signal_generate
  157. # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter
  158. -bash: echo: write error: Invalid argument
  159. # cat filter
  160. ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash
  161. ^
  162. parse_error: Field not found
  163. Currently the caret ('^') for an error always appears at the beginning of
  164. the filter string; the error message should still be useful though
  165. even without more accurate position info.
  166. 5.2.1 Filter limitations
  167. ------------------------
  168. If a filter is placed on a string pointer ``(char *)`` that does not point
  169. to a string on the ring buffer, but instead points to kernel or user space
  170. memory, then, for safety reasons, at most 1024 bytes of the content is
  171. copied onto a temporary buffer to do the compare. If the copy of the memory
  172. faults (the pointer points to memory that should not be accessed), then the
  173. string compare will be treated as not matching.
  174. 5.3 Clearing filters
  175. --------------------
  176. To clear the filter for an event, write a '0' to the event's filter
  177. file.
  178. To clear the filters for all events in a subsystem, write a '0' to the
  179. subsystem's filter file.
  180. 5.3 Subsystem filters
  181. ---------------------
  182. For convenience, filters for every event in a subsystem can be set or
  183. cleared as a group by writing a filter expression into the filter file
  184. at the root of the subsystem. Note however, that if a filter for any
  185. event within the subsystem lacks a field specified in the subsystem
  186. filter, or if the filter can't be applied for any other reason, the
  187. filter for that event will retain its previous setting. This can
  188. result in an unintended mixture of filters which could lead to
  189. confusing (to the user who might think different filters are in
  190. effect) trace output. Only filters that reference just the common
  191. fields can be guaranteed to propagate successfully to all events.
  192. Here are a few subsystem filter examples that also illustrate the
  193. above points:
  194. Clear the filters on all events in the sched subsystem::
  195. # cd /sys/kernel/debug/tracing/events/sched
  196. # echo 0 > filter
  197. # cat sched_switch/filter
  198. none
  199. # cat sched_wakeup/filter
  200. none
  201. Set a filter using only common fields for all events in the sched
  202. subsystem (all events end up with the same filter)::
  203. # cd /sys/kernel/debug/tracing/events/sched
  204. # echo common_pid == 0 > filter
  205. # cat sched_switch/filter
  206. common_pid == 0
  207. # cat sched_wakeup/filter
  208. common_pid == 0
  209. Attempt to set a filter using a non-common field for all events in the
  210. sched subsystem (all events but those that have a prev_pid field retain
  211. their old filters)::
  212. # cd /sys/kernel/debug/tracing/events/sched
  213. # echo prev_pid == 0 > filter
  214. # cat sched_switch/filter
  215. prev_pid == 0
  216. # cat sched_wakeup/filter
  217. common_pid == 0
  218. 5.4 PID filtering
  219. -----------------
  220. The set_event_pid file in the same directory as the top events directory
  221. exists, will filter all events from tracing any task that does not have the
  222. PID listed in the set_event_pid file.
  223. ::
  224. # cd /sys/kernel/debug/tracing
  225. # echo $$ > set_event_pid
  226. # echo 1 > events/enable
  227. Will only trace events for the current task.
  228. To add more PIDs without losing the PIDs already included, use '>>'.
  229. ::
  230. # echo 123 244 1 >> set_event_pid
  231. 6. Event triggers
  232. =================
  233. Trace events can be made to conditionally invoke trigger 'commands'
  234. which can take various forms and are described in detail below;
  235. examples would be enabling or disabling other trace events or invoking
  236. a stack trace whenever the trace event is hit. Whenever a trace event
  237. with attached triggers is invoked, the set of trigger commands
  238. associated with that event is invoked. Any given trigger can
  239. additionally have an event filter of the same form as described in
  240. section 5 (Event filtering) associated with it - the command will only
  241. be invoked if the event being invoked passes the associated filter.
  242. If no filter is associated with the trigger, it always passes.
  243. Triggers are added to and removed from a particular event by writing
  244. trigger expressions to the 'trigger' file for the given event.
  245. A given event can have any number of triggers associated with it,
  246. subject to any restrictions that individual commands may have in that
  247. regard.
  248. Event triggers are implemented on top of "soft" mode, which means that
  249. whenever a trace event has one or more triggers associated with it,
  250. the event is activated even if it isn't actually enabled, but is
  251. disabled in a "soft" mode. That is, the tracepoint will be called,
  252. but just will not be traced, unless of course it's actually enabled.
  253. This scheme allows triggers to be invoked even for events that aren't
  254. enabled, and also allows the current event filter implementation to be
  255. used for conditionally invoking triggers.
  256. The syntax for event triggers is roughly based on the syntax for
  257. set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands'
  258. section of Documentation/trace/ftrace.rst), but there are major
  259. differences and the implementation isn't currently tied to it in any
  260. way, so beware about making generalizations between the two.
  261. .. Note::
  262. Writing into trace_marker (See Documentation/trace/ftrace.rst)
  263. can also enable triggers that are written into
  264. /sys/kernel/tracing/events/ftrace/print/trigger
  265. 6.1 Expression syntax
  266. ---------------------
  267. Triggers are added by echoing the command to the 'trigger' file::
  268. # echo 'command[:count] [if filter]' > trigger
  269. Triggers are removed by echoing the same command but starting with '!'
  270. to the 'trigger' file::
  271. # echo '!command[:count] [if filter]' > trigger
  272. The [if filter] part isn't used in matching commands when removing, so
  273. leaving that off in a '!' command will accomplish the same thing as
  274. having it in.
  275. The filter syntax is the same as that described in the 'Event
  276. filtering' section above.
  277. For ease of use, writing to the trigger file using '>' currently just
  278. adds or removes a single trigger and there's no explicit '>>' support
  279. ('>' actually behaves like '>>') or truncation support to remove all
  280. triggers (you have to use '!' for each one added.)
  281. 6.2 Supported trigger commands
  282. ------------------------------
  283. The following commands are supported:
  284. - enable_event/disable_event
  285. These commands can enable or disable another trace event whenever
  286. the triggering event is hit. When these commands are registered,
  287. the other trace event is activated, but disabled in a "soft" mode.
  288. That is, the tracepoint will be called, but just will not be traced.
  289. The event tracepoint stays in this mode as long as there's a trigger
  290. in effect that can trigger it.
  291. For example, the following trigger causes kmalloc events to be
  292. traced when a read system call is entered, and the :1 at the end
  293. specifies that this enablement happens only once::
  294. # echo 'enable_event:kmem:kmalloc:1' > \
  295. /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger
  296. The following trigger causes kmalloc events to stop being traced
  297. when a read system call exits. This disablement happens on every
  298. read system call exit::
  299. # echo 'disable_event:kmem:kmalloc' > \
  300. /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger
  301. The format is::
  302. enable_event:<system>:<event>[:count]
  303. disable_event:<system>:<event>[:count]
  304. To remove the above commands::
  305. # echo '!enable_event:kmem:kmalloc:1' > \
  306. /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger
  307. # echo '!disable_event:kmem:kmalloc' > \
  308. /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger
  309. Note that there can be any number of enable/disable_event triggers
  310. per triggering event, but there can only be one trigger per
  311. triggered event. e.g. sys_enter_read can have triggers enabling both
  312. kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc
  313. versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if
  314. bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they
  315. could be combined into a single filter on kmem:kmalloc though).
  316. - stacktrace
  317. This command dumps a stacktrace in the trace buffer whenever the
  318. triggering event occurs.
  319. For example, the following trigger dumps a stacktrace every time the
  320. kmalloc tracepoint is hit::
  321. # echo 'stacktrace' > \
  322. /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
  323. The following trigger dumps a stacktrace the first 5 times a kmalloc
  324. request happens with a size >= 64K::
  325. # echo 'stacktrace:5 if bytes_req >= 65536' > \
  326. /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
  327. The format is::
  328. stacktrace[:count]
  329. To remove the above commands::
  330. # echo '!stacktrace' > \
  331. /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
  332. # echo '!stacktrace:5 if bytes_req >= 65536' > \
  333. /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
  334. The latter can also be removed more simply by the following (without
  335. the filter)::
  336. # echo '!stacktrace:5' > \
  337. /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
  338. Note that there can be only one stacktrace trigger per triggering
  339. event.
  340. - snapshot
  341. This command causes a snapshot to be triggered whenever the
  342. triggering event occurs.
  343. The following command creates a snapshot every time a block request
  344. queue is unplugged with a depth > 1. If you were tracing a set of
  345. events or functions at the time, the snapshot trace buffer would
  346. capture those events when the trigger event occurred::
  347. # echo 'snapshot if nr_rq > 1' > \
  348. /sys/kernel/debug/tracing/events/block/block_unplug/trigger
  349. To only snapshot once::
  350. # echo 'snapshot:1 if nr_rq > 1' > \
  351. /sys/kernel/debug/tracing/events/block/block_unplug/trigger
  352. To remove the above commands::
  353. # echo '!snapshot if nr_rq > 1' > \
  354. /sys/kernel/debug/tracing/events/block/block_unplug/trigger
  355. # echo '!snapshot:1 if nr_rq > 1' > \
  356. /sys/kernel/debug/tracing/events/block/block_unplug/trigger
  357. Note that there can be only one snapshot trigger per triggering
  358. event.
  359. - traceon/traceoff
  360. These commands turn tracing on and off when the specified events are
  361. hit. The parameter determines how many times the tracing system is
  362. turned on and off. If unspecified, there is no limit.
  363. The following command turns tracing off the first time a block
  364. request queue is unplugged with a depth > 1. If you were tracing a
  365. set of events or functions at the time, you could then examine the
  366. trace buffer to see the sequence of events that led up to the
  367. trigger event::
  368. # echo 'traceoff:1 if nr_rq > 1' > \
  369. /sys/kernel/debug/tracing/events/block/block_unplug/trigger
  370. To always disable tracing when nr_rq > 1::
  371. # echo 'traceoff if nr_rq > 1' > \
  372. /sys/kernel/debug/tracing/events/block/block_unplug/trigger
  373. To remove the above commands::
  374. # echo '!traceoff:1 if nr_rq > 1' > \
  375. /sys/kernel/debug/tracing/events/block/block_unplug/trigger
  376. # echo '!traceoff if nr_rq > 1' > \
  377. /sys/kernel/debug/tracing/events/block/block_unplug/trigger
  378. Note that there can be only one traceon or traceoff trigger per
  379. triggering event.
  380. - hist
  381. This command aggregates event hits into a hash table keyed on one or
  382. more trace event format fields (or stacktrace) and a set of running
  383. totals derived from one or more trace event format fields and/or
  384. event counts (hitcount).
  385. See Documentation/trace/histogram.rst for details and examples.
  386. 7. In-kernel trace event API
  387. ============================
  388. In most cases, the command-line interface to trace events is more than
  389. sufficient. Sometimes, however, applications might find the need for
  390. more complex relationships than can be expressed through a simple
  391. series of linked command-line expressions, or putting together sets of
  392. commands may be simply too cumbersome. An example might be an
  393. application that needs to 'listen' to the trace stream in order to
  394. maintain an in-kernel state machine detecting, for instance, when an
  395. illegal kernel state occurs in the scheduler.
  396. The trace event subsystem provides an in-kernel API allowing modules
  397. or other kernel code to generate user-defined 'synthetic' events at
  398. will, which can be used to either augment the existing trace stream
  399. and/or signal that a particular important state has occurred.
  400. A similar in-kernel API is also available for creating kprobe and
  401. kretprobe events.
  402. Both the synthetic event and k/ret/probe event APIs are built on top
  403. of a lower-level "dynevent_cmd" event command API, which is also
  404. available for more specialized applications, or as the basis of other
  405. higher-level trace event APIs.
  406. The API provided for these purposes is describe below and allows the
  407. following:
  408. - dynamically creating synthetic event definitions
  409. - dynamically creating kprobe and kretprobe event definitions
  410. - tracing synthetic events from in-kernel code
  411. - the low-level "dynevent_cmd" API
  412. 7.1 Dyamically creating synthetic event definitions
  413. ---------------------------------------------------
  414. There are a couple ways to create a new synthetic event from a kernel
  415. module or other kernel code.
  416. The first creates the event in one step, using synth_event_create().
  417. In this method, the name of the event to create and an array defining
  418. the fields is supplied to synth_event_create(). If successful, a
  419. synthetic event with that name and fields will exist following that
  420. call. For example, to create a new "schedtest" synthetic event::
  421. ret = synth_event_create("schedtest", sched_fields,
  422. ARRAY_SIZE(sched_fields), THIS_MODULE);
  423. The sched_fields param in this example points to an array of struct
  424. synth_field_desc, each of which describes an event field by type and
  425. name::
  426. static struct synth_field_desc sched_fields[] = {
  427. { .type = "pid_t", .name = "next_pid_field" },
  428. { .type = "char[16]", .name = "next_comm_field" },
  429. { .type = "u64", .name = "ts_ns" },
  430. { .type = "u64", .name = "ts_ms" },
  431. { .type = "unsigned int", .name = "cpu" },
  432. { .type = "char[64]", .name = "my_string_field" },
  433. { .type = "int", .name = "my_int_field" },
  434. };
  435. See synth_field_size() for available types.
  436. If field_name contains [n], the field is considered to be a static array.
  437. If field_names contains[] (no subscript), the field is considered to
  438. be a dynamic array, which will only take as much space in the event as
  439. is required to hold the array.
  440. Because space for an event is reserved before assigning field values
  441. to the event, using dynamic arrays implies that the piecewise
  442. in-kernel API described below can't be used with dynamic arrays. The
  443. other non-piecewise in-kernel APIs can, however, be used with dynamic
  444. arrays.
  445. If the event is created from within a module, a pointer to the module
  446. must be passed to synth_event_create(). This will ensure that the
  447. trace buffer won't contain unreadable events when the module is
  448. removed.
  449. At this point, the event object is ready to be used for generating new
  450. events.
  451. In the second method, the event is created in several steps. This
  452. allows events to be created dynamically and without the need to create
  453. and populate an array of fields beforehand.
  454. To use this method, an empty or partially empty synthetic event should
  455. first be created using synth_event_gen_cmd_start() or
  456. synth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(),
  457. the name of the event along with one or more pairs of args each pair
  458. representing a 'type field_name;' field specification should be
  459. supplied. For synth_event_gen_cmd_array_start(), the name of the
  460. event along with an array of struct synth_field_desc should be
  461. supplied. Before calling synth_event_gen_cmd_start() or
  462. synth_event_gen_cmd_array_start(), the user should create and
  463. initialize a dynevent_cmd object using synth_event_cmd_init().
  464. For example, to create a new "schedtest" synthetic event with two
  465. fields::
  466. struct dynevent_cmd cmd;
  467. char *buf;
  468. /* Create a buffer to hold the generated command */
  469. buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
  470. /* Before generating the command, initialize the cmd object */
  471. synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
  472. ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE,
  473. "pid_t", "next_pid_field",
  474. "u64", "ts_ns");
  475. Alternatively, using an array of struct synth_field_desc fields
  476. containing the same information::
  477. ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE,
  478. fields, n_fields);
  479. Once the synthetic event object has been created, it can then be
  480. populated with more fields. Fields are added one by one using
  481. synth_event_add_field(), supplying the dynevent_cmd object, a field
  482. type, and a field name. For example, to add a new int field named
  483. "intfield", the following call should be made::
  484. ret = synth_event_add_field(&cmd, "int", "intfield");
  485. See synth_field_size() for available types. If field_name contains [n]
  486. the field is considered to be an array.
  487. A group of fields can also be added all at once using an array of
  488. synth_field_desc with add_synth_fields(). For example, this would add
  489. just the first four sched_fields::
  490. ret = synth_event_add_fields(&cmd, sched_fields, 4);
  491. If you already have a string of the form 'type field_name',
  492. synth_event_add_field_str() can be used to add it as-is; it will
  493. also automatically append a ';' to the string.
  494. Once all the fields have been added, the event should be finalized and
  495. registered by calling the synth_event_gen_cmd_end() function::
  496. ret = synth_event_gen_cmd_end(&cmd);
  497. At this point, the event object is ready to be used for tracing new
  498. events.
  499. 7.2 Tracing synthetic events from in-kernel code
  500. ------------------------------------------------
  501. To trace a synthetic event, there are several options. The first
  502. option is to trace the event in one call, using synth_event_trace()
  503. with a variable number of values, or synth_event_trace_array() with an
  504. array of values to be set. A second option can be used to avoid the
  505. need for a pre-formed array of values or list of arguments, via
  506. synth_event_trace_start() and synth_event_trace_end() along with
  507. synth_event_add_next_val() or synth_event_add_val() to add the values
  508. piecewise.
  509. 7.2.1 Tracing a synthetic event all at once
  510. -------------------------------------------
  511. To trace a synthetic event all at once, the synth_event_trace() or
  512. synth_event_trace_array() functions can be used.
  513. The synth_event_trace() function is passed the trace_event_file
  514. representing the synthetic event (which can be retrieved using
  515. trace_get_event_file() using the synthetic event name, "synthetic" as
  516. the system name, and the trace instance name (NULL if using the global
  517. trace array)), along with an variable number of u64 args, one for each
  518. synthetic event field, and the number of values being passed.
  519. So, to trace an event corresponding to the synthetic event definition
  520. above, code like the following could be used::
  521. ret = synth_event_trace(create_synth_test, 7, /* number of values */
  522. 444, /* next_pid_field */
  523. (u64)"clackers", /* next_comm_field */
  524. 1000000, /* ts_ns */
  525. 1000, /* ts_ms */
  526. smp_processor_id(),/* cpu */
  527. (u64)"Thneed", /* my_string_field */
  528. 999); /* my_int_field */
  529. All vals should be cast to u64, and string vals are just pointers to
  530. strings, cast to u64. Strings will be copied into space reserved in
  531. the event for the string, using these pointers.
  532. Alternatively, the synth_event_trace_array() function can be used to
  533. accomplish the same thing. It is passed the trace_event_file
  534. representing the synthetic event (which can be retrieved using
  535. trace_get_event_file() using the synthetic event name, "synthetic" as
  536. the system name, and the trace instance name (NULL if using the global
  537. trace array)), along with an array of u64, one for each synthetic
  538. event field.
  539. To trace an event corresponding to the synthetic event definition
  540. above, code like the following could be used::
  541. u64 vals[7];
  542. vals[0] = 777; /* next_pid_field */
  543. vals[1] = (u64)"tiddlywinks"; /* next_comm_field */
  544. vals[2] = 1000000; /* ts_ns */
  545. vals[3] = 1000; /* ts_ms */
  546. vals[4] = smp_processor_id(); /* cpu */
  547. vals[5] = (u64)"thneed"; /* my_string_field */
  548. vals[6] = 398; /* my_int_field */
  549. The 'vals' array is just an array of u64, the number of which must
  550. match the number of field in the synthetic event, and which must be in
  551. the same order as the synthetic event fields.
  552. All vals should be cast to u64, and string vals are just pointers to
  553. strings, cast to u64. Strings will be copied into space reserved in
  554. the event for the string, using these pointers.
  555. In order to trace a synthetic event, a pointer to the trace event file
  556. is needed. The trace_get_event_file() function can be used to get
  557. it - it will find the file in the given trace instance (in this case
  558. NULL since the top trace array is being used) while at the same time
  559. preventing the instance containing it from going away::
  560. schedtest_event_file = trace_get_event_file(NULL, "synthetic",
  561. "schedtest");
  562. Before tracing the event, it should be enabled in some way, otherwise
  563. the synthetic event won't actually show up in the trace buffer.
  564. To enable a synthetic event from the kernel, trace_array_set_clr_event()
  565. can be used (which is not specific to synthetic events, so does need
  566. the "synthetic" system name to be specified explicitly).
  567. To enable the event, pass 'true' to it::
  568. trace_array_set_clr_event(schedtest_event_file->tr,
  569. "synthetic", "schedtest", true);
  570. To disable it pass false::
  571. trace_array_set_clr_event(schedtest_event_file->tr,
  572. "synthetic", "schedtest", false);
  573. Finally, synth_event_trace_array() can be used to actually trace the
  574. event, which should be visible in the trace buffer afterwards::
  575. ret = synth_event_trace_array(schedtest_event_file, vals,
  576. ARRAY_SIZE(vals));
  577. To remove the synthetic event, the event should be disabled, and the
  578. trace instance should be 'put' back using trace_put_event_file()::
  579. trace_array_set_clr_event(schedtest_event_file->tr,
  580. "synthetic", "schedtest", false);
  581. trace_put_event_file(schedtest_event_file);
  582. If those have been successful, synth_event_delete() can be called to
  583. remove the event::
  584. ret = synth_event_delete("schedtest");
  585. 7.2.2 Tracing a synthetic event piecewise
  586. -----------------------------------------
  587. To trace a synthetic using the piecewise method described above, the
  588. synth_event_trace_start() function is used to 'open' the synthetic
  589. event trace::
  590. struct synth_trace_state trace_state;
  591. ret = synth_event_trace_start(schedtest_event_file, &trace_state);
  592. It's passed the trace_event_file representing the synthetic event
  593. using the same methods as described above, along with a pointer to a
  594. struct synth_trace_state object, which will be zeroed before use and
  595. used to maintain state between this and following calls.
  596. Once the event has been opened, which means space for it has been
  597. reserved in the trace buffer, the individual fields can be set. There
  598. are two ways to do that, either one after another for each field in
  599. the event, which requires no lookups, or by name, which does. The
  600. tradeoff is flexibility in doing the assignments vs the cost of a
  601. lookup per field.
  602. To assign the values one after the other without lookups,
  603. synth_event_add_next_val() should be used. Each call is passed the
  604. same synth_trace_state object used in the synth_event_trace_start(),
  605. along with the value to set the next field in the event. After each
  606. field is set, the 'cursor' points to the next field, which will be set
  607. by the subsequent call, continuing until all the fields have been set
  608. in order. The same sequence of calls as in the above examples using
  609. this method would be (without error-handling code)::
  610. /* next_pid_field */
  611. ret = synth_event_add_next_val(777, &trace_state);
  612. /* next_comm_field */
  613. ret = synth_event_add_next_val((u64)"slinky", &trace_state);
  614. /* ts_ns */
  615. ret = synth_event_add_next_val(1000000, &trace_state);
  616. /* ts_ms */
  617. ret = synth_event_add_next_val(1000, &trace_state);
  618. /* cpu */
  619. ret = synth_event_add_next_val(smp_processor_id(), &trace_state);
  620. /* my_string_field */
  621. ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state);
  622. /* my_int_field */
  623. ret = synth_event_add_next_val(395, &trace_state);
  624. To assign the values in any order, synth_event_add_val() should be
  625. used. Each call is passed the same synth_trace_state object used in
  626. the synth_event_trace_start(), along with the field name of the field
  627. to set and the value to set it to. The same sequence of calls as in
  628. the above examples using this method would be (without error-handling
  629. code)::
  630. ret = synth_event_add_val("next_pid_field", 777, &trace_state);
  631. ret = synth_event_add_val("next_comm_field", (u64)"silly putty",
  632. &trace_state);
  633. ret = synth_event_add_val("ts_ns", 1000000, &trace_state);
  634. ret = synth_event_add_val("ts_ms", 1000, &trace_state);
  635. ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state);
  636. ret = synth_event_add_val("my_string_field", (u64)"thneed_9",
  637. &trace_state);
  638. ret = synth_event_add_val("my_int_field", 3999, &trace_state);
  639. Note that synth_event_add_next_val() and synth_event_add_val() are
  640. incompatible if used within the same trace of an event - either one
  641. can be used but not both at the same time.
  642. Finally, the event won't be actually traced until it's 'closed',
  643. which is done using synth_event_trace_end(), which takes only the
  644. struct synth_trace_state object used in the previous calls::
  645. ret = synth_event_trace_end(&trace_state);
  646. Note that synth_event_trace_end() must be called at the end regardless
  647. of whether any of the add calls failed (say due to a bad field name
  648. being passed in).
  649. 7.3 Dyamically creating kprobe and kretprobe event definitions
  650. --------------------------------------------------------------
  651. To create a kprobe or kretprobe trace event from kernel code, the
  652. kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start()
  653. functions can be used.
  654. To create a kprobe event, an empty or partially empty kprobe event
  655. should first be created using kprobe_event_gen_cmd_start(). The name
  656. of the event and the probe location should be specfied along with one
  657. or args each representing a probe field should be supplied to this
  658. function. Before calling kprobe_event_gen_cmd_start(), the user
  659. should create and initialize a dynevent_cmd object using
  660. kprobe_event_cmd_init().
  661. For example, to create a new "schedtest" kprobe event with two fields::
  662. struct dynevent_cmd cmd;
  663. char *buf;
  664. /* Create a buffer to hold the generated command */
  665. buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
  666. /* Before generating the command, initialize the cmd object */
  667. kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
  668. /*
  669. * Define the gen_kprobe_test event with the first 2 kprobe
  670. * fields.
  671. */
  672. ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open",
  673. "dfd=%ax", "filename=%dx");
  674. Once the kprobe event object has been created, it can then be
  675. populated with more fields. Fields can be added using
  676. kprobe_event_add_fields(), supplying the dynevent_cmd object along
  677. with a variable arg list of probe fields. For example, to add a
  678. couple additional fields, the following call could be made::
  679. ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");
  680. Once all the fields have been added, the event should be finalized and
  681. registered by calling the kprobe_event_gen_cmd_end() or
  682. kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe
  683. or kretprobe command was started::
  684. ret = kprobe_event_gen_cmd_end(&cmd);
  685. or::
  686. ret = kretprobe_event_gen_cmd_end(&cmd);
  687. At this point, the event object is ready to be used for tracing new
  688. events.
  689. Similarly, a kretprobe event can be created using
  690. kretprobe_event_gen_cmd_start() with a probe name and location and
  691. additional params such as $retval::
  692. ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test",
  693. "do_sys_open", "$retval");
  694. Similar to the synthetic event case, code like the following can be
  695. used to enable the newly created kprobe event::
  696. gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test");
  697. ret = trace_array_set_clr_event(gen_kprobe_test->tr,
  698. "kprobes", "gen_kprobe_test", true);
  699. Finally, also similar to synthetic events, the following code can be
  700. used to give the kprobe event file back and delete the event::
  701. trace_put_event_file(gen_kprobe_test);
  702. ret = kprobe_event_delete("gen_kprobe_test");
  703. 7.4 The "dynevent_cmd" low-level API
  704. ------------------------------------
  705. Both the in-kernel synthetic event and kprobe interfaces are built on
  706. top of a lower-level "dynevent_cmd" interface. This interface is
  707. meant to provide the basis for higher-level interfaces such as the
  708. synthetic and kprobe interfaces, which can be used as examples.
  709. The basic idea is simple and amounts to providing a general-purpose
  710. layer that can be used to generate trace event commands. The
  711. generated command strings can then be passed to the command-parsing
  712. and event creation code that already exists in the trace event
  713. subystem for creating the corresponding trace events.
  714. In a nutshell, the way it works is that the higher-level interface
  715. code creates a struct dynevent_cmd object, then uses a couple
  716. functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up
  717. a command string, which finally causes the command to be executed
  718. using the dynevent_create() function. The details of the interface
  719. are described below.
  720. The first step in building a new command string is to create and
  721. initialize an instance of a dynevent_cmd. Here, for instance, we
  722. create a dynevent_cmd on the stack and initialize it::
  723. struct dynevent_cmd cmd;
  724. char *buf;
  725. int ret;
  726. buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
  727. dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO,
  728. foo_event_run_command);
  729. The dynevent_cmd initialization needs to be given a user-specified
  730. buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used
  731. for this purpose - at 2k it's generally too big to be comfortably put
  732. on the stack, so is dynamically allocated), a dynevent type id, which
  733. is meant to be used to check that further API calls are for the
  734. correct command type, and a pointer to an event-specific run_command()
  735. callback that will be called to actually execute the event-specific
  736. command function.
  737. Once that's done, the command string can by built up by successive
  738. calls to argument-adding functions.
  739. To add a single argument, define and initialize a struct dynevent_arg
  740. or struct dynevent_arg_pair object. Here's an example of the simplest
  741. possible arg addition, which is simply to append the given string as
  742. a whitespace-separated argument to the command::
  743. struct dynevent_arg arg;
  744. dynevent_arg_init(&arg, NULL, 0);
  745. arg.str = name;
  746. ret = dynevent_arg_add(cmd, &arg);
  747. The arg object is first initialized using dynevent_arg_init() and in
  748. this case the parameters are NULL or 0, which means there's no
  749. optional sanity-checking function or separator appended to the end of
  750. the arg.
  751. Here's another more complicated example using an 'arg pair', which is
  752. used to create an argument that consists of a couple components added
  753. together as a unit, for example, a 'type field_name;' arg or a simple
  754. expression arg e.g. 'flags=%cx'::
  755. struct dynevent_arg_pair arg_pair;
  756. dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';');
  757. arg_pair.lhs = type;
  758. arg_pair.rhs = name;
  759. ret = dynevent_arg_pair_add(cmd, &arg_pair);
  760. Again, the arg_pair is first initialized, in this case with a callback
  761. function used to check the sanity of the args (for example, that
  762. neither part of the pair is NULL), along with a character to be used
  763. to add an operator between the pair (here none) and a separator to be
  764. appended onto the end of the arg pair (here ';').
  765. There's also a dynevent_str_add() function that can be used to simply
  766. add a string as-is, with no spaces, delimeters, or arg check.
  767. Any number of dynevent_*_add() calls can be made to build up the string
  768. (until its length surpasses cmd->maxlen). When all the arguments have
  769. been added and the command string is complete, the only thing left to
  770. do is run the command, which happens by simply calling
  771. dynevent_create()::
  772. ret = dynevent_create(&cmd);
  773. At that point, if the return value is 0, the dynamic event has been
  774. created and is ready to use.
  775. See the dynevent_cmd function definitions themselves for the details
  776. of the API.