timestamping.rst 34 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756
  1. .. SPDX-License-Identifier: GPL-2.0
  2. ============
  3. Timestamping
  4. ============
  5. 1. Control Interfaces
  6. =====================
  7. The interfaces for receiving network packages timestamps are:
  8. SO_TIMESTAMP
  9. Generates a timestamp for each incoming packet in (not necessarily
  10. monotonic) system time. Reports the timestamp via recvmsg() in a
  11. control message in usec resolution.
  12. SO_TIMESTAMP is defined as SO_TIMESTAMP_NEW or SO_TIMESTAMP_OLD
  13. based on the architecture type and time_t representation of libc.
  14. Control message format is in struct __kernel_old_timeval for
  15. SO_TIMESTAMP_OLD and in struct __kernel_sock_timeval for
  16. SO_TIMESTAMP_NEW options respectively.
  17. SO_TIMESTAMPNS
  18. Same timestamping mechanism as SO_TIMESTAMP, but reports the
  19. timestamp as struct timespec in nsec resolution.
  20. SO_TIMESTAMPNS is defined as SO_TIMESTAMPNS_NEW or SO_TIMESTAMPNS_OLD
  21. based on the architecture type and time_t representation of libc.
  22. Control message format is in struct timespec for SO_TIMESTAMPNS_OLD
  23. and in struct __kernel_timespec for SO_TIMESTAMPNS_NEW options
  24. respectively.
  25. IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]
  26. Only for multicast:approximate transmit timestamp obtained by
  27. reading the looped packet receive timestamp.
  28. SO_TIMESTAMPING
  29. Generates timestamps on reception, transmission or both. Supports
  30. multiple timestamp sources, including hardware. Supports generating
  31. timestamps for stream sockets.
  32. 1.1 SO_TIMESTAMP (also SO_TIMESTAMP_OLD and SO_TIMESTAMP_NEW)
  33. -------------------------------------------------------------
  34. This socket option enables timestamping of datagrams on the reception
  35. path. Because the destination socket, if any, is not known early in
  36. the network stack, the feature has to be enabled for all packets. The
  37. same is true for all early receive timestamp options.
  38. For interface details, see `man 7 socket`.
  39. Always use SO_TIMESTAMP_NEW timestamp to always get timestamp in
  40. struct __kernel_sock_timeval format.
  41. SO_TIMESTAMP_OLD returns incorrect timestamps after the year 2038
  42. on 32 bit machines.
  43. 1.2 SO_TIMESTAMPNS (also SO_TIMESTAMPNS_OLD and SO_TIMESTAMPNS_NEW):
  44. This option is identical to SO_TIMESTAMP except for the returned data type.
  45. Its struct timespec allows for higher resolution (ns) timestamps than the
  46. timeval of SO_TIMESTAMP (ms).
  47. Always use SO_TIMESTAMPNS_NEW timestamp to always get timestamp in
  48. struct __kernel_timespec format.
  49. SO_TIMESTAMPNS_OLD returns incorrect timestamps after the year 2038
  50. on 32 bit machines.
  51. 1.3 SO_TIMESTAMPING (also SO_TIMESTAMPING_OLD and SO_TIMESTAMPING_NEW)
  52. ----------------------------------------------------------------------
  53. Supports multiple types of timestamp requests. As a result, this
  54. socket option takes a bitmap of flags, not a boolean. In::
  55. err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));
  56. val is an integer with any of the following bits set. Setting other
  57. bit returns EINVAL and does not change the current state.
  58. The socket option configures timestamp generation for individual
  59. sk_buffs (1.3.1), timestamp reporting to the socket's error
  60. queue (1.3.2) and options (1.3.3). Timestamp generation can also
  61. be enabled for individual sendmsg calls using cmsg (1.3.4).
  62. 1.3.1 Timestamp Generation
  63. ^^^^^^^^^^^^^^^^^^^^^^^^^^
  64. Some bits are requests to the stack to try to generate timestamps. Any
  65. combination of them is valid. Changes to these bits apply to newly
  66. created packets, not to packets already in the stack. As a result, it
  67. is possible to selectively request timestamps for a subset of packets
  68. (e.g., for sampling) by embedding an send() call within two setsockopt
  69. calls, one to enable timestamp generation and one to disable it.
  70. Timestamps may also be generated for reasons other than being
  71. requested by a particular socket, such as when receive timestamping is
  72. enabled system wide, as explained earlier.
  73. SOF_TIMESTAMPING_RX_HARDWARE:
  74. Request rx timestamps generated by the network adapter.
  75. SOF_TIMESTAMPING_RX_SOFTWARE:
  76. Request rx timestamps when data enters the kernel. These timestamps
  77. are generated just after a device driver hands a packet to the
  78. kernel receive stack.
  79. SOF_TIMESTAMPING_TX_HARDWARE:
  80. Request tx timestamps generated by the network adapter. This flag
  81. can be enabled via both socket options and control messages.
  82. SOF_TIMESTAMPING_TX_SOFTWARE:
  83. Request tx timestamps when data leaves the kernel. These timestamps
  84. are generated in the device driver as close as possible, but always
  85. prior to, passing the packet to the network interface. Hence, they
  86. require driver support and may not be available for all devices.
  87. This flag can be enabled via both socket options and control messages.
  88. SOF_TIMESTAMPING_TX_SCHED:
  89. Request tx timestamps prior to entering the packet scheduler. Kernel
  90. transmit latency is, if long, often dominated by queuing delay. The
  91. difference between this timestamp and one taken at
  92. SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent
  93. of protocol processing. The latency incurred in protocol
  94. processing, if any, can be computed by subtracting a userspace
  95. timestamp taken immediately before send() from this timestamp. On
  96. machines with virtual devices where a transmitted packet travels
  97. through multiple devices and, hence, multiple packet schedulers,
  98. a timestamp is generated at each layer. This allows for fine
  99. grained measurement of queuing delay. This flag can be enabled
  100. via both socket options and control messages.
  101. SOF_TIMESTAMPING_TX_ACK:
  102. Request tx timestamps when all data in the send buffer has been
  103. acknowledged. This only makes sense for reliable protocols. It is
  104. currently only implemented for TCP. For that protocol, it may
  105. over-report measurement, because the timestamp is generated when all
  106. data up to and including the buffer at send() was acknowledged: the
  107. cumulative acknowledgment. The mechanism ignores SACK and FACK.
  108. This flag can be enabled via both socket options and control messages.
  109. 1.3.2 Timestamp Reporting
  110. ^^^^^^^^^^^^^^^^^^^^^^^^^
  111. The other three bits control which timestamps will be reported in a
  112. generated control message. Changes to the bits take immediate
  113. effect at the timestamp reporting locations in the stack. Timestamps
  114. are only reported for packets that also have the relevant timestamp
  115. generation request set.
  116. SOF_TIMESTAMPING_SOFTWARE:
  117. Report any software timestamps when available.
  118. SOF_TIMESTAMPING_SYS_HARDWARE:
  119. This option is deprecated and ignored.
  120. SOF_TIMESTAMPING_RAW_HARDWARE:
  121. Report hardware timestamps as generated by
  122. SOF_TIMESTAMPING_TX_HARDWARE when available.
  123. 1.3.3 Timestamp Options
  124. ^^^^^^^^^^^^^^^^^^^^^^^
  125. The interface supports the options
  126. SOF_TIMESTAMPING_OPT_ID:
  127. Generate a unique identifier along with each packet. A process can
  128. have multiple concurrent timestamping requests outstanding. Packets
  129. can be reordered in the transmit path, for instance in the packet
  130. scheduler. In that case timestamps will be queued onto the error
  131. queue out of order from the original send() calls. It is not always
  132. possible to uniquely match timestamps to the original send() calls
  133. based on timestamp order or payload inspection alone, then.
  134. This option associates each packet at send() with a unique
  135. identifier and returns that along with the timestamp. The identifier
  136. is derived from a per-socket u32 counter (that wraps). For datagram
  137. sockets, the counter increments with each sent packet. For stream
  138. sockets, it increments with every byte.
  139. The counter starts at zero. It is initialized the first time that
  140. the socket option is enabled. It is reset each time the option is
  141. enabled after having been disabled. Resetting the counter does not
  142. change the identifiers of existing packets in the system.
  143. This option is implemented only for transmit timestamps. There, the
  144. timestamp is always looped along with a struct sock_extended_err.
  145. The option modifies field ee_data to pass an id that is unique
  146. among all possibly concurrently outstanding timestamp requests for
  147. that socket.
  148. SOF_TIMESTAMPING_OPT_CMSG:
  149. Support recv() cmsg for all timestamped packets. Control messages
  150. are already supported unconditionally on all packets with receive
  151. timestamps and on IPv6 packets with transmit timestamp. This option
  152. extends them to IPv4 packets with transmit timestamp. One use case
  153. is to correlate packets with their egress device, by enabling socket
  154. option IP_PKTINFO simultaneously.
  155. SOF_TIMESTAMPING_OPT_TSONLY:
  156. Applies to transmit timestamps only. Makes the kernel return the
  157. timestamp as a cmsg alongside an empty packet, as opposed to
  158. alongside the original packet. This reduces the amount of memory
  159. charged to the socket's receive budget (SO_RCVBUF) and delivers
  160. the timestamp even if sysctl net.core.tstamp_allow_data is 0.
  161. This option disables SOF_TIMESTAMPING_OPT_CMSG.
  162. SOF_TIMESTAMPING_OPT_STATS:
  163. Optional stats that are obtained along with the transmit timestamps.
  164. It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When the
  165. transmit timestamp is available, the stats are available in a
  166. separate control message of type SCM_TIMESTAMPING_OPT_STATS, as a
  167. list of TLVs (struct nlattr) of types. These stats allow the
  168. application to associate various transport layer stats with
  169. the transmit timestamps, such as how long a certain block of
  170. data was limited by peer's receiver window.
  171. SOF_TIMESTAMPING_OPT_PKTINFO:
  172. Enable the SCM_TIMESTAMPING_PKTINFO control message for incoming
  173. packets with hardware timestamps. The message contains struct
  174. scm_ts_pktinfo, which supplies the index of the real interface which
  175. received the packet and its length at layer 2. A valid (non-zero)
  176. interface index will be returned only if CONFIG_NET_RX_BUSY_POLL is
  177. enabled and the driver is using NAPI. The struct contains also two
  178. other fields, but they are reserved and undefined.
  179. SOF_TIMESTAMPING_OPT_TX_SWHW:
  180. Request both hardware and software timestamps for outgoing packets
  181. when SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE
  182. are enabled at the same time. If both timestamps are generated,
  183. two separate messages will be looped to the socket's error queue,
  184. each containing just one timestamp.
  185. New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to
  186. disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate
  187. regardless of the setting of sysctl net.core.tstamp_allow_data.
  188. An exception is when a process needs additional cmsg data, for
  189. instance SOL_IP/IP_PKTINFO to detect the egress network interface.
  190. Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on
  191. having access to the contents of the original packet, so cannot be
  192. combined with SOF_TIMESTAMPING_OPT_TSONLY.
  193. 1.3.4. Enabling timestamps via control messages
  194. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  195. In addition to socket options, timestamp generation can be requested
  196. per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1).
  197. Using this feature, applications can sample timestamps per sendmsg()
  198. without paying the overhead of enabling and disabling timestamps via
  199. setsockopt::
  200. struct msghdr *msg;
  201. ...
  202. cmsg = CMSG_FIRSTHDR(msg);
  203. cmsg->cmsg_level = SOL_SOCKET;
  204. cmsg->cmsg_type = SO_TIMESTAMPING;
  205. cmsg->cmsg_len = CMSG_LEN(sizeof(__u32));
  206. *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED |
  207. SOF_TIMESTAMPING_TX_SOFTWARE |
  208. SOF_TIMESTAMPING_TX_ACK;
  209. err = sendmsg(fd, msg, 0);
  210. The SOF_TIMESTAMPING_TX_* flags set via cmsg will override
  211. the SOF_TIMESTAMPING_TX_* flags set via setsockopt.
  212. Moreover, applications must still enable timestamp reporting via
  213. setsockopt to receive timestamps::
  214. __u32 val = SOF_TIMESTAMPING_SOFTWARE |
  215. SOF_TIMESTAMPING_OPT_ID /* or any other flag */;
  216. err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));
  217. 1.4 Bytestream Timestamps
  218. -------------------------
  219. The SO_TIMESTAMPING interface supports timestamping of bytes in a
  220. bytestream. Each request is interpreted as a request for when the
  221. entire contents of the buffer has passed a timestamping point. That
  222. is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record
  223. when all bytes have reached the device driver, regardless of how
  224. many packets the data has been converted into.
  225. In general, bytestreams have no natural delimiters and therefore
  226. correlating a timestamp with data is non-trivial. A range of bytes
  227. may be split across segments, any segments may be merged (possibly
  228. coalescing sections of previously segmented buffers associated with
  229. independent send() calls). Segments can be reordered and the same
  230. byte range can coexist in multiple segments for protocols that
  231. implement retransmissions.
  232. It is essential that all timestamps implement the same semantics,
  233. regardless of these possible transformations, as otherwise they are
  234. incomparable. Handling "rare" corner cases differently from the
  235. simple case (a 1:1 mapping from buffer to skb) is insufficient
  236. because performance debugging often needs to focus on such outliers.
  237. In practice, timestamps can be correlated with segments of a
  238. bytestream consistently, if both semantics of the timestamp and the
  239. timing of measurement are chosen correctly. This challenge is no
  240. different from deciding on a strategy for IP fragmentation. There, the
  241. definition is that only the first fragment is timestamped. For
  242. bytestreams, we chose that a timestamp is generated only when all
  243. bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to
  244. implement and reason about. An implementation that has to take into
  245. account SACK would be more complex due to possible transmission holes
  246. and out of order arrival.
  247. On the host, TCP can also break the simple 1:1 mapping from buffer to
  248. skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The
  249. implementation ensures correctness in all cases by tracking the
  250. individual last byte passed to send(), even if it is no longer the
  251. last byte after an skbuff extend or merge operation. It stores the
  252. relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff
  253. has only one such field, only one timestamp can be generated.
  254. In rare cases, a timestamp request can be missed if two requests are
  255. collapsed onto the same skb. A process can detect this situation by
  256. enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at
  257. send time with the value returned for each timestamp. It can prevent
  258. the situation by always flushing the TCP stack in between requests,
  259. for instance by enabling TCP_NODELAY and disabling TCP_CORK and
  260. autocork.
  261. These precautions ensure that the timestamp is generated only when all
  262. bytes have passed a timestamp point, assuming that the network stack
  263. itself does not reorder the segments. The stack indeed tries to avoid
  264. reordering. The one exception is under administrator control: it is
  265. possible to construct a packet scheduler configuration that delays
  266. segments from the same stream differently. Such a setup would be
  267. unusual.
  268. 2 Data Interfaces
  269. ==================
  270. Timestamps are read using the ancillary data feature of recvmsg().
  271. See `man 3 cmsg` for details of this interface. The socket manual
  272. page (`man 7 socket`) describes how timestamps generated with
  273. SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.
  274. 2.1 SCM_TIMESTAMPING records
  275. ----------------------------
  276. These timestamps are returned in a control message with cmsg_level
  277. SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type
  278. For SO_TIMESTAMPING_OLD::
  279. struct scm_timestamping {
  280. struct timespec ts[3];
  281. };
  282. For SO_TIMESTAMPING_NEW::
  283. struct scm_timestamping64 {
  284. struct __kernel_timespec ts[3];
  285. Always use SO_TIMESTAMPING_NEW timestamp to always get timestamp in
  286. struct scm_timestamping64 format.
  287. SO_TIMESTAMPING_OLD returns incorrect timestamps after the year 2038
  288. on 32 bit machines.
  289. The structure can return up to three timestamps. This is a legacy
  290. feature. At least one field is non-zero at any time. Most timestamps
  291. are passed in ts[0]. Hardware timestamps are passed in ts[2].
  292. ts[1] used to hold hardware timestamps converted to system time.
  293. Instead, expose the hardware clock device on the NIC directly as
  294. a HW PTP clock source, to allow time conversion in userspace and
  295. optionally synchronize system time with a userspace PTP stack such
  296. as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst.
  297. Note that if the SO_TIMESTAMP or SO_TIMESTAMPNS option is enabled
  298. together with SO_TIMESTAMPING using SOF_TIMESTAMPING_SOFTWARE, a false
  299. software timestamp will be generated in the recvmsg() call and passed
  300. in ts[0] when a real software timestamp is missing. This happens also
  301. on hardware transmit timestamps.
  302. 2.1.1 Transmit timestamps with MSG_ERRQUEUE
  303. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  304. For transmit timestamps the outgoing packet is looped back to the
  305. socket's error queue with the send timestamp(s) attached. A process
  306. receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE
  307. set and with a msg_control buffer sufficiently large to receive the
  308. relevant metadata structures. The recvmsg call returns the original
  309. outgoing data packet with two ancillary messages attached.
  310. A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR
  311. embeds a struct sock_extended_err. This defines the error type. For
  312. timestamps, the ee_errno field is ENOMSG. The other ancillary message
  313. will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This
  314. embeds the struct scm_timestamping.
  315. 2.1.1.2 Timestamp types
  316. ~~~~~~~~~~~~~~~~~~~~~~~
  317. The semantics of the three struct timespec are defined by field
  318. ee_info in the extended error structure. It contains a value of
  319. type SCM_TSTAMP_* to define the actual timestamp passed in
  320. scm_timestamping.
  321. The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*
  322. control fields discussed previously, with one exception. For legacy
  323. reasons, SCM_TSTAMP_SND is equal to zero and can be set for both
  324. SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It
  325. is the first if ts[2] is non-zero, the second otherwise, in which
  326. case the timestamp is stored in ts[0].
  327. 2.1.1.3 Fragmentation
  328. ~~~~~~~~~~~~~~~~~~~~~
  329. Fragmentation of outgoing datagrams is rare, but is possible, e.g., by
  330. explicitly disabling PMTU discovery. If an outgoing packet is fragmented,
  331. then only the first fragment is timestamped and returned to the sending
  332. socket.
  333. 2.1.1.4 Packet Payload
  334. ~~~~~~~~~~~~~~~~~~~~~~
  335. The calling application is often not interested in receiving the whole
  336. packet payload that it passed to the stack originally: the socket
  337. error queue mechanism is just a method to piggyback the timestamp on.
  338. In this case, the application can choose to read datagrams with a
  339. smaller buffer, possibly even of length 0. The payload is truncated
  340. accordingly. Until the process calls recvmsg() on the error queue,
  341. however, the full packet is queued, taking up budget from SO_RCVBUF.
  342. 2.1.1.5 Blocking Read
  343. ~~~~~~~~~~~~~~~~~~~~~
  344. Reading from the error queue is always a non-blocking operation. To
  345. block waiting on a timestamp, use poll or select. poll() will return
  346. POLLERR in pollfd.revents if any data is ready on the error queue.
  347. There is no need to pass this flag in pollfd.events. This flag is
  348. ignored on request. See also `man 2 poll`.
  349. 2.1.2 Receive timestamps
  350. ^^^^^^^^^^^^^^^^^^^^^^^^
  351. On reception, there is no reason to read from the socket error queue.
  352. The SCM_TIMESTAMPING ancillary data is sent along with the packet data
  353. on a normal recvmsg(). Since this is not a socket error, it is not
  354. accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,
  355. the meaning of the three fields in struct scm_timestamping is
  356. implicitly defined. ts[0] holds a software timestamp if set, ts[1]
  357. is again deprecated and ts[2] holds a hardware timestamp if set.
  358. 3. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP
  359. =======================================================================
  360. Hardware time stamping must also be initialized for each device driver
  361. that is expected to do hardware time stamping. The parameter is defined in
  362. include/uapi/linux/net_tstamp.h as::
  363. struct hwtstamp_config {
  364. int flags; /* no flags defined right now, must be zero */
  365. int tx_type; /* HWTSTAMP_TX_* */
  366. int rx_filter; /* HWTSTAMP_FILTER_* */
  367. };
  368. Desired behavior is passed into the kernel and to a specific device by
  369. calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose
  370. ifr_data points to a struct hwtstamp_config. The tx_type and
  371. rx_filter are hints to the driver what it is expected to do. If
  372. the requested fine-grained filtering for incoming packets is not
  373. supported, the driver may time stamp more than just the requested types
  374. of packets.
  375. Drivers are free to use a more permissive configuration than the requested
  376. configuration. It is expected that drivers should only implement directly the
  377. most generic mode that can be supported. For example if the hardware can
  378. support HWTSTAMP_FILTER_V2_EVENT, then it should generally always upscale
  379. HWTSTAMP_FILTER_V2_L2_SYNC_MESSAGE, and so forth, as HWTSTAMP_FILTER_V2_EVENT
  380. is more generic (and more useful to applications).
  381. A driver which supports hardware time stamping shall update the struct
  382. with the actual, possibly more permissive configuration. If the
  383. requested packets cannot be time stamped, then nothing should be
  384. changed and ERANGE shall be returned (in contrast to EINVAL, which
  385. indicates that SIOCSHWTSTAMP is not supported at all).
  386. Only a processes with admin rights may change the configuration. User
  387. space is responsible to ensure that multiple processes don't interfere
  388. with each other and that the settings are reset.
  389. Any process can read the actual configuration by passing this
  390. structure to ioctl(SIOCGHWTSTAMP) in the same way. However, this has
  391. not been implemented in all drivers.
  392. ::
  393. /* possible values for hwtstamp_config->tx_type */
  394. enum {
  395. /*
  396. * no outgoing packet will need hardware time stamping;
  397. * should a packet arrive which asks for it, no hardware
  398. * time stamping will be done
  399. */
  400. HWTSTAMP_TX_OFF,
  401. /*
  402. * enables hardware time stamping for outgoing packets;
  403. * the sender of the packet decides which are to be
  404. * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE
  405. * before sending the packet
  406. */
  407. HWTSTAMP_TX_ON,
  408. };
  409. /* possible values for hwtstamp_config->rx_filter */
  410. enum {
  411. /* time stamp no incoming packet at all */
  412. HWTSTAMP_FILTER_NONE,
  413. /* time stamp any incoming packet */
  414. HWTSTAMP_FILTER_ALL,
  415. /* return value: time stamp all packets requested plus some others */
  416. HWTSTAMP_FILTER_SOME,
  417. /* PTP v1, UDP, any kind of event packet */
  418. HWTSTAMP_FILTER_PTP_V1_L4_EVENT,
  419. /* for the complete list of values, please check
  420. * the include file include/uapi/linux/net_tstamp.h
  421. */
  422. };
  423. 3.1 Hardware Timestamping Implementation: Device Drivers
  424. --------------------------------------------------------
  425. A driver which supports hardware time stamping must support the
  426. SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with
  427. the actual values as described in the section on SIOCSHWTSTAMP. It
  428. should also support SIOCGHWTSTAMP.
  429. Time stamps for received packets must be stored in the skb. To get a pointer
  430. to the shared time stamp structure of the skb call skb_hwtstamps(). Then
  431. set the time stamps in the structure::
  432. struct skb_shared_hwtstamps {
  433. /* hardware time stamp transformed into duration
  434. * since arbitrary point in time
  435. */
  436. ktime_t hwtstamp;
  437. };
  438. Time stamps for outgoing packets are to be generated as follows:
  439. - In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)
  440. is set no-zero. If yes, then the driver is expected to do hardware time
  441. stamping.
  442. - If this is possible for the skb and requested, then declare
  443. that the driver is doing the time stamping by setting the flag
  444. SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with::
  445. skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
  446. You might want to keep a pointer to the associated skb for the next step
  447. and not free the skb. A driver not supporting hardware time stamping doesn't
  448. do that. A driver must never touch sk_buff::tstamp! It is used to store
  449. software generated time stamps by the network subsystem.
  450. - Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware
  451. as possible. skb_tx_timestamp() provides a software time stamp if requested
  452. and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).
  453. - As soon as the driver has sent the packet and/or obtained a
  454. hardware time stamp for it, it passes the time stamp back by
  455. calling skb_hwtstamp_tx() with the original skb, the raw
  456. hardware time stamp. skb_hwtstamp_tx() clones the original skb and
  457. adds the timestamps, therefore the original skb has to be freed now.
  458. If obtaining the hardware time stamp somehow fails, then the driver
  459. should not fall back to software time stamping. The rationale is that
  460. this would occur at a later time in the processing pipeline than other
  461. software time stamping and therefore could lead to unexpected deltas
  462. between time stamps.
  463. 3.2 Special considerations for stacked PTP Hardware Clocks
  464. ----------------------------------------------------------
  465. There are situations when there may be more than one PHC (PTP Hardware Clock)
  466. in the data path of a packet. The kernel has no explicit mechanism to allow the
  467. user to select which PHC to use for timestamping Ethernet frames. Instead, the
  468. assumption is that the outermost PHC is always the most preferable, and that
  469. kernel drivers collaborate towards achieving that goal. Currently there are 3
  470. cases of stacked PHCs, detailed below:
  471. 3.2.1 DSA (Distributed Switch Architecture) switches
  472. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  473. These are Ethernet switches which have one of their ports connected to an
  474. (otherwise completely unaware) host Ethernet interface, and perform the role of
  475. a port multiplier with optional forwarding acceleration features. Each DSA
  476. switch port is visible to the user as a standalone (virtual) network interface,
  477. and its network I/O is performed, under the hood, indirectly through the host
  478. interface (redirecting to the host port on TX, and intercepting frames on RX).
  479. When a DSA switch is attached to a host port, PTP synchronization has to
  480. suffer, since the switch's variable queuing delay introduces a path delay
  481. jitter between the host port and its PTP partner. For this reason, some DSA
  482. switches include a timestamping clock of their own, and have the ability to
  483. perform network timestamping on their own MAC, such that path delays only
  484. measure wire and PHY propagation latencies. Timestamping DSA switches are
  485. supported in Linux and expose the same ABI as any other network interface (save
  486. for the fact that the DSA interfaces are in fact virtual in terms of network
  487. I/O, they do have their own PHC). It is typical, but not mandatory, for all
  488. interfaces of a DSA switch to share the same PHC.
  489. By design, PTP timestamping with a DSA switch does not need any special
  490. handling in the driver for the host port it is attached to. However, when the
  491. host port also supports PTP timestamping, DSA will take care of intercepting
  492. the ``.ndo_do_ioctl`` calls towards the host port, and block attempts to enable
  493. hardware timestamping on it. This is because the SO_TIMESTAMPING API does not
  494. allow the delivery of multiple hardware timestamps for the same packet, so
  495. anybody else except for the DSA switch port must be prevented from doing so.
  496. In code, DSA provides for most of the infrastructure for timestamping already,
  497. in generic code: a BPF classifier (``ptp_classify_raw``) is used to identify
  498. PTP event messages (any other packets, including PTP general messages, are not
  499. timestamped), and provides two hooks to drivers:
  500. - ``.port_txtstamp()``: The driver is passed a clone of the timestampable skb
  501. to be transmitted, before actually transmitting it. Typically, a switch will
  502. have a PTP TX timestamp register (or sometimes a FIFO) where the timestamp
  503. becomes available. There may be an IRQ that is raised upon this timestamp's
  504. availability, or the driver might have to poll after invoking
  505. ``dev_queue_xmit()`` towards the host interface. Either way, in the
  506. ``.port_txtstamp()`` method, the driver only needs to save the clone for
  507. later use (when the timestamp becomes available). Each skb is annotated with
  508. a pointer to its clone, in ``DSA_SKB_CB(skb)->clone``, to ease the driver's
  509. job of keeping track of which clone belongs to which skb.
  510. - ``.port_rxtstamp()``: The original (and only) timestampable skb is provided
  511. to the driver, for it to annotate it with a timestamp, if that is immediately
  512. available, or defer to later. On reception, timestamps might either be
  513. available in-band (through metadata in the DSA header, or attached in other
  514. ways to the packet), or out-of-band (through another RX timestamping FIFO).
  515. Deferral on RX is typically necessary when retrieving the timestamp needs a
  516. sleepable context. In that case, it is the responsibility of the DSA driver
  517. to call ``netif_rx_ni()`` on the freshly timestamped skb.
  518. 3.2.2 Ethernet PHYs
  519. ^^^^^^^^^^^^^^^^^^^
  520. These are devices that typically fulfill a Layer 1 role in the network stack,
  521. hence they do not have a representation in terms of a network interface as DSA
  522. switches do. However, PHYs may be able to detect and timestamp PTP packets, for
  523. performance reasons: timestamps taken as close as possible to the wire have the
  524. potential to yield a more stable and precise synchronization.
  525. A PHY driver that supports PTP timestamping must create a ``struct
  526. mii_timestamper`` and add a pointer to it in ``phydev->mii_ts``. The presence
  527. of this pointer will be checked by the networking stack.
  528. Since PHYs do not have network interface representations, the timestamping and
  529. ethtool ioctl operations for them need to be mediated by their respective MAC
  530. driver. Therefore, as opposed to DSA switches, modifications need to be done
  531. to each individual MAC driver for PHY timestamping support. This entails:
  532. - Checking, in ``.ndo_do_ioctl``, whether ``phy_has_hwtstamp(netdev->phydev)``
  533. is true or not. If it is, then the MAC driver should not process this request
  534. but instead pass it on to the PHY using ``phy_mii_ioctl()``.
  535. - On RX, special intervention may or may not be needed, depending on the
  536. function used to deliver skb's up the network stack. In the case of plain
  537. ``netif_rx()`` and similar, MAC drivers must check whether
  538. ``skb_defer_rx_timestamp(skb)`` is necessary or not - and if it is, don't
  539. call ``netif_rx()`` at all. If ``CONFIG_NETWORK_PHY_TIMESTAMPING`` is
  540. enabled, and ``skb->dev->phydev->mii_ts`` exists, its ``.rxtstamp()`` hook
  541. will be called now, to determine, using logic very similar to DSA, whether
  542. deferral for RX timestamping is necessary. Again like DSA, it becomes the
  543. responsibility of the PHY driver to send the packet up the stack when the
  544. timestamp is available.
  545. For other skb receive functions, such as ``napi_gro_receive`` and
  546. ``netif_receive_skb``, the stack automatically checks whether
  547. ``skb_defer_rx_timestamp()`` is necessary, so this check is not needed inside
  548. the driver.
  549. - On TX, again, special intervention might or might not be needed. The
  550. function that calls the ``mii_ts->txtstamp()`` hook is named
  551. ``skb_clone_tx_timestamp()``. This function can either be called directly
  552. (case in which explicit MAC driver support is indeed needed), but the
  553. function also piggybacks from the ``skb_tx_timestamp()`` call, which many MAC
  554. drivers already perform for software timestamping purposes. Therefore, if a
  555. MAC supports software timestamping, it does not need to do anything further
  556. at this stage.
  557. 3.2.3 MII bus snooping devices
  558. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  559. These perform the same role as timestamping Ethernet PHYs, save for the fact
  560. that they are discrete devices and can therefore be used in conjunction with
  561. any PHY even if it doesn't support timestamping. In Linux, they are
  562. discoverable and attachable to a ``struct phy_device`` through Device Tree, and
  563. for the rest, they use the same mii_ts infrastructure as those. See
  564. Documentation/devicetree/bindings/ptp/timestamper.txt for more details.
  565. 3.2.4 Other caveats for MAC drivers
  566. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  567. Stacked PHCs, especially DSA (but not only) - since that doesn't require any
  568. modification to MAC drivers, so it is more difficult to ensure correctness of
  569. all possible code paths - is that they uncover bugs which were impossible to
  570. trigger before the existence of stacked PTP clocks. One example has to do with
  571. this line of code, already presented earlier::
  572. skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
  573. Any TX timestamping logic, be it a plain MAC driver, a DSA switch driver, a PHY
  574. driver or a MII bus snooping device driver, should set this flag.
  575. But a MAC driver that is unaware of PHC stacking might get tripped up by
  576. somebody other than itself setting this flag, and deliver a duplicate
  577. timestamp.
  578. For example, a typical driver design for TX timestamping might be to split the
  579. transmission part into 2 portions:
  580. 1. "TX": checks whether PTP timestamping has been previously enabled through
  581. the ``.ndo_do_ioctl`` ("``priv->hwtstamp_tx_enabled == true``") and the
  582. current skb requires a TX timestamp ("``skb_shinfo(skb)->tx_flags &
  583. SKBTX_HW_TSTAMP``"). If this is true, it sets the
  584. "``skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS``" flag. Note: as
  585. described above, in the case of a stacked PHC system, this condition should
  586. never trigger, as this MAC is certainly not the outermost PHC. But this is
  587. not where the typical issue is. Transmission proceeds with this packet.
  588. 2. "TX confirmation": Transmission has finished. The driver checks whether it
  589. is necessary to collect any TX timestamp for it. Here is where the typical
  590. issues are: the MAC driver takes a shortcut and only checks whether
  591. "``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``" was set. With a stacked
  592. PHC system, this is incorrect because this MAC driver is not the only entity
  593. in the TX data path who could have enabled SKBTX_IN_PROGRESS in the first
  594. place.
  595. The correct solution for this problem is for MAC drivers to have a compound
  596. check in their "TX confirmation" portion, not only for
  597. "``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``", but also for
  598. "``priv->hwtstamp_tx_enabled == true``". Because the rest of the system ensures
  599. that PTP timestamping is not enabled for anything other than the outermost PHC,
  600. this enhanced check will avoid delivering a duplicated TX timestamp to user
  601. space.