multi-touch-protocol.rst 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411
  1. .. include:: <isonum.txt>
  2. =========================
  3. Multi-touch (MT) Protocol
  4. =========================
  5. :Copyright: |copy| 2009-2010 Henrik Rydberg <rydberg@euromail.se>
  6. Introduction
  7. ------------
  8. In order to utilize the full power of the new multi-touch and multi-user
  9. devices, a way to report detailed data from multiple contacts, i.e.,
  10. objects in direct contact with the device surface, is needed. This
  11. document describes the multi-touch (MT) protocol which allows kernel
  12. drivers to report details for an arbitrary number of contacts.
  13. The protocol is divided into two types, depending on the capabilities of the
  14. hardware. For devices handling anonymous contacts (type A), the protocol
  15. describes how to send the raw data for all contacts to the receiver. For
  16. devices capable of tracking identifiable contacts (type B), the protocol
  17. describes how to send updates for individual contacts via event slots.
  18. .. note::
  19. MT protocol type A is obsolete, all kernel drivers have been
  20. converted to use type B.
  21. Protocol Usage
  22. --------------
  23. Contact details are sent sequentially as separate packets of ABS_MT
  24. events. Only the ABS_MT events are recognized as part of a contact
  25. packet. Since these events are ignored by current single-touch (ST)
  26. applications, the MT protocol can be implemented on top of the ST protocol
  27. in an existing driver.
  28. Drivers for type A devices separate contact packets by calling
  29. input_mt_sync() at the end of each packet. This generates a SYN_MT_REPORT
  30. event, which instructs the receiver to accept the data for the current
  31. contact and prepare to receive another.
  32. Drivers for type B devices separate contact packets by calling
  33. input_mt_slot(), with a slot as argument, at the beginning of each packet.
  34. This generates an ABS_MT_SLOT event, which instructs the receiver to
  35. prepare for updates of the given slot.
  36. All drivers mark the end of a multi-touch transfer by calling the usual
  37. input_sync() function. This instructs the receiver to act upon events
  38. accumulated since last EV_SYN/SYN_REPORT and prepare to receive a new set
  39. of events/packets.
  40. The main difference between the stateless type A protocol and the stateful
  41. type B slot protocol lies in the usage of identifiable contacts to reduce
  42. the amount of data sent to userspace. The slot protocol requires the use of
  43. the ABS_MT_TRACKING_ID, either provided by the hardware or computed from
  44. the raw data [#f5]_.
  45. For type A devices, the kernel driver should generate an arbitrary
  46. enumeration of the full set of anonymous contacts currently on the
  47. surface. The order in which the packets appear in the event stream is not
  48. important. Event filtering and finger tracking is left to user space [#f3]_.
  49. For type B devices, the kernel driver should associate a slot with each
  50. identified contact, and use that slot to propagate changes for the contact.
  51. Creation, replacement and destruction of contacts is achieved by modifying
  52. the ABS_MT_TRACKING_ID of the associated slot. A non-negative tracking id
  53. is interpreted as a contact, and the value -1 denotes an unused slot. A
  54. tracking id not previously present is considered new, and a tracking id no
  55. longer present is considered removed. Since only changes are propagated,
  56. the full state of each initiated contact has to reside in the receiving
  57. end. Upon receiving an MT event, one simply updates the appropriate
  58. attribute of the current slot.
  59. Some devices identify and/or track more contacts than they can report to the
  60. driver. A driver for such a device should associate one type B slot with each
  61. contact that is reported by the hardware. Whenever the identity of the
  62. contact associated with a slot changes, the driver should invalidate that
  63. slot by changing its ABS_MT_TRACKING_ID. If the hardware signals that it is
  64. tracking more contacts than it is currently reporting, the driver should use
  65. a BTN_TOOL_*TAP event to inform userspace of the total number of contacts
  66. being tracked by the hardware at that moment. The driver should do this by
  67. explicitly sending the corresponding BTN_TOOL_*TAP event and setting
  68. use_count to false when calling input_mt_report_pointer_emulation().
  69. The driver should only advertise as many slots as the hardware can report.
  70. Userspace can detect that a driver can report more total contacts than slots
  71. by noting that the largest supported BTN_TOOL_*TAP event is larger than the
  72. total number of type B slots reported in the absinfo for the ABS_MT_SLOT axis.
  73. The minimum value of the ABS_MT_SLOT axis must be 0.
  74. Protocol Example A
  75. ------------------
  76. Here is what a minimal event sequence for a two-contact touch would look
  77. like for a type A device::
  78. ABS_MT_POSITION_X x[0]
  79. ABS_MT_POSITION_Y y[0]
  80. SYN_MT_REPORT
  81. ABS_MT_POSITION_X x[1]
  82. ABS_MT_POSITION_Y y[1]
  83. SYN_MT_REPORT
  84. SYN_REPORT
  85. The sequence after moving one of the contacts looks exactly the same; the
  86. raw data for all present contacts are sent between every synchronization
  87. with SYN_REPORT.
  88. Here is the sequence after lifting the first contact::
  89. ABS_MT_POSITION_X x[1]
  90. ABS_MT_POSITION_Y y[1]
  91. SYN_MT_REPORT
  92. SYN_REPORT
  93. And here is the sequence after lifting the second contact::
  94. SYN_MT_REPORT
  95. SYN_REPORT
  96. If the driver reports one of BTN_TOUCH or ABS_PRESSURE in addition to the
  97. ABS_MT events, the last SYN_MT_REPORT event may be omitted. Otherwise, the
  98. last SYN_REPORT will be dropped by the input core, resulting in no
  99. zero-contact event reaching userland.
  100. Protocol Example B
  101. ------------------
  102. Here is what a minimal event sequence for a two-contact touch would look
  103. like for a type B device::
  104. ABS_MT_SLOT 0
  105. ABS_MT_TRACKING_ID 45
  106. ABS_MT_POSITION_X x[0]
  107. ABS_MT_POSITION_Y y[0]
  108. ABS_MT_SLOT 1
  109. ABS_MT_TRACKING_ID 46
  110. ABS_MT_POSITION_X x[1]
  111. ABS_MT_POSITION_Y y[1]
  112. SYN_REPORT
  113. Here is the sequence after moving contact 45 in the x direction::
  114. ABS_MT_SLOT 0
  115. ABS_MT_POSITION_X x[0]
  116. SYN_REPORT
  117. Here is the sequence after lifting the contact in slot 0::
  118. ABS_MT_TRACKING_ID -1
  119. SYN_REPORT
  120. The slot being modified is already 0, so the ABS_MT_SLOT is omitted. The
  121. message removes the association of slot 0 with contact 45, thereby
  122. destroying contact 45 and freeing slot 0 to be reused for another contact.
  123. Finally, here is the sequence after lifting the second contact::
  124. ABS_MT_SLOT 1
  125. ABS_MT_TRACKING_ID -1
  126. SYN_REPORT
  127. Event Usage
  128. -----------
  129. A set of ABS_MT events with the desired properties is defined. The events
  130. are divided into categories, to allow for partial implementation. The
  131. minimum set consists of ABS_MT_POSITION_X and ABS_MT_POSITION_Y, which
  132. allows for multiple contacts to be tracked. If the device supports it, the
  133. ABS_MT_TOUCH_MAJOR and ABS_MT_WIDTH_MAJOR may be used to provide the size
  134. of the contact area and approaching tool, respectively.
  135. The TOUCH and WIDTH parameters have a geometrical interpretation; imagine
  136. looking through a window at someone gently holding a finger against the
  137. glass. You will see two regions, one inner region consisting of the part
  138. of the finger actually touching the glass, and one outer region formed by
  139. the perimeter of the finger. The center of the touching region (a) is
  140. ABS_MT_POSITION_X/Y and the center of the approaching finger (b) is
  141. ABS_MT_TOOL_X/Y. The touch diameter is ABS_MT_TOUCH_MAJOR and the finger
  142. diameter is ABS_MT_WIDTH_MAJOR. Now imagine the person pressing the finger
  143. harder against the glass. The touch region will increase, and in general,
  144. the ratio ABS_MT_TOUCH_MAJOR / ABS_MT_WIDTH_MAJOR, which is always smaller
  145. than unity, is related to the contact pressure. For pressure-based devices,
  146. ABS_MT_PRESSURE may be used to provide the pressure on the contact area
  147. instead. Devices capable of contact hovering can use ABS_MT_DISTANCE to
  148. indicate the distance between the contact and the surface.
  149. ::
  150. Linux MT Win8
  151. __________ _______________________
  152. / \ | |
  153. / \ | |
  154. / ____ \ | |
  155. / / \ \ | |
  156. \ \ a \ \ | a |
  157. \ \____/ \ | |
  158. \ \ | |
  159. \ b \ | b |
  160. \ \ | |
  161. \ \ | |
  162. \ \ | |
  163. \ / | |
  164. \ / | |
  165. \ / | |
  166. \__________/ |_______________________|
  167. In addition to the MAJOR parameters, the oval shape of the touch and finger
  168. regions can be described by adding the MINOR parameters, such that MAJOR
  169. and MINOR are the major and minor axis of an ellipse. The orientation of
  170. the touch ellipse can be described with the ORIENTATION parameter, and the
  171. direction of the finger ellipse is given by the vector (a - b).
  172. For type A devices, further specification of the touch shape is possible
  173. via ABS_MT_BLOB_ID.
  174. The ABS_MT_TOOL_TYPE may be used to specify whether the touching tool is a
  175. finger or a pen or something else. Finally, the ABS_MT_TRACKING_ID event
  176. may be used to track identified contacts over time [#f5]_.
  177. In the type B protocol, ABS_MT_TOOL_TYPE and ABS_MT_TRACKING_ID are
  178. implicitly handled by input core; drivers should instead call
  179. input_mt_report_slot_state().
  180. Event Semantics
  181. ---------------
  182. ABS_MT_TOUCH_MAJOR
  183. The length of the major axis of the contact. The length should be given in
  184. surface units. If the surface has an X times Y resolution, the largest
  185. possible value of ABS_MT_TOUCH_MAJOR is sqrt(X^2 + Y^2), the diagonal [#f4]_.
  186. ABS_MT_TOUCH_MINOR
  187. The length, in surface units, of the minor axis of the contact. If the
  188. contact is circular, this event can be omitted [#f4]_.
  189. ABS_MT_WIDTH_MAJOR
  190. The length, in surface units, of the major axis of the approaching
  191. tool. This should be understood as the size of the tool itself. The
  192. orientation of the contact and the approaching tool are assumed to be the
  193. same [#f4]_.
  194. ABS_MT_WIDTH_MINOR
  195. The length, in surface units, of the minor axis of the approaching
  196. tool. Omit if circular [#f4]_.
  197. The above four values can be used to derive additional information about
  198. the contact. The ratio ABS_MT_TOUCH_MAJOR / ABS_MT_WIDTH_MAJOR approximates
  199. the notion of pressure. The fingers of the hand and the palm all have
  200. different characteristic widths.
  201. ABS_MT_PRESSURE
  202. The pressure, in arbitrary units, on the contact area. May be used instead
  203. of TOUCH and WIDTH for pressure-based devices or any device with a spatial
  204. signal intensity distribution.
  205. ABS_MT_DISTANCE
  206. The distance, in surface units, between the contact and the surface. Zero
  207. distance means the contact is touching the surface. A positive number means
  208. the contact is hovering above the surface.
  209. ABS_MT_ORIENTATION
  210. The orientation of the touching ellipse. The value should describe a signed
  211. quarter of a revolution clockwise around the touch center. The signed value
  212. range is arbitrary, but zero should be returned for an ellipse aligned with
  213. the Y axis (north) of the surface, a negative value when the ellipse is
  214. turned to the left, and a positive value when the ellipse is turned to the
  215. right. When aligned with the X axis in the positive direction, the range
  216. max should be returned; when aligned with the X axis in the negative
  217. direction, the range -max should be returned.
  218. Touch ellipsis are symmetrical by default. For devices capable of true 360
  219. degree orientation, the reported orientation must exceed the range max to
  220. indicate more than a quarter of a revolution. For an upside-down finger,
  221. range max * 2 should be returned.
  222. Orientation can be omitted if the touch area is circular, or if the
  223. information is not available in the kernel driver. Partial orientation
  224. support is possible if the device can distinguish between the two axis, but
  225. not (uniquely) any values in between. In such cases, the range of
  226. ABS_MT_ORIENTATION should be [0, 1] [#f4]_.
  227. ABS_MT_POSITION_X
  228. The surface X coordinate of the center of the touching ellipse.
  229. ABS_MT_POSITION_Y
  230. The surface Y coordinate of the center of the touching ellipse.
  231. ABS_MT_TOOL_X
  232. The surface X coordinate of the center of the approaching tool. Omit if
  233. the device cannot distinguish between the intended touch point and the
  234. tool itself.
  235. ABS_MT_TOOL_Y
  236. The surface Y coordinate of the center of the approaching tool. Omit if the
  237. device cannot distinguish between the intended touch point and the tool
  238. itself.
  239. The four position values can be used to separate the position of the touch
  240. from the position of the tool. If both positions are present, the major
  241. tool axis points towards the touch point [#f1]_. Otherwise, the tool axes are
  242. aligned with the touch axes.
  243. ABS_MT_TOOL_TYPE
  244. The type of approaching tool. A lot of kernel drivers cannot distinguish
  245. between different tool types, such as a finger or a pen. In such cases, the
  246. event should be omitted. The protocol currently mainly supports
  247. MT_TOOL_FINGER, MT_TOOL_PEN, and MT_TOOL_PALM [#f2]_.
  248. For type B devices, this event is handled by input core; drivers should
  249. instead use input_mt_report_slot_state(). A contact's ABS_MT_TOOL_TYPE may
  250. change over time while still touching the device, because the firmware may
  251. not be able to determine which tool is being used when it first appears.
  252. ABS_MT_BLOB_ID
  253. The BLOB_ID groups several packets together into one arbitrarily shaped
  254. contact. The sequence of points forms a polygon which defines the shape of
  255. the contact. This is a low-level anonymous grouping for type A devices, and
  256. should not be confused with the high-level trackingID [#f5]_. Most type A
  257. devices do not have blob capability, so drivers can safely omit this event.
  258. ABS_MT_TRACKING_ID
  259. The TRACKING_ID identifies an initiated contact throughout its life cycle
  260. [#f5]_. The value range of the TRACKING_ID should be large enough to ensure
  261. unique identification of a contact maintained over an extended period of
  262. time. For type B devices, this event is handled by input core; drivers
  263. should instead use input_mt_report_slot_state().
  264. Event Computation
  265. -----------------
  266. The flora of different hardware unavoidably leads to some devices fitting
  267. better to the MT protocol than others. To simplify and unify the mapping,
  268. this section gives recipes for how to compute certain events.
  269. For devices reporting contacts as rectangular shapes, signed orientation
  270. cannot be obtained. Assuming X and Y are the lengths of the sides of the
  271. touching rectangle, here is a simple formula that retains the most
  272. information possible::
  273. ABS_MT_TOUCH_MAJOR := max(X, Y)
  274. ABS_MT_TOUCH_MINOR := min(X, Y)
  275. ABS_MT_ORIENTATION := bool(X > Y)
  276. The range of ABS_MT_ORIENTATION should be set to [0, 1], to indicate that
  277. the device can distinguish between a finger along the Y axis (0) and a
  278. finger along the X axis (1).
  279. For win8 devices with both T and C coordinates, the position mapping is::
  280. ABS_MT_POSITION_X := T_X
  281. ABS_MT_POSITION_Y := T_Y
  282. ABS_MT_TOOL_X := C_X
  283. ABS_MT_TOOL_Y := C_Y
  284. Unfortunately, there is not enough information to specify both the touching
  285. ellipse and the tool ellipse, so one has to resort to approximations. One
  286. simple scheme, which is compatible with earlier usage, is::
  287. ABS_MT_TOUCH_MAJOR := min(X, Y)
  288. ABS_MT_TOUCH_MINOR := <not used>
  289. ABS_MT_ORIENTATION := <not used>
  290. ABS_MT_WIDTH_MAJOR := min(X, Y) + distance(T, C)
  291. ABS_MT_WIDTH_MINOR := min(X, Y)
  292. Rationale: We have no information about the orientation of the touching
  293. ellipse, so approximate it with an inscribed circle instead. The tool
  294. ellipse should align with the vector (T - C), so the diameter must
  295. increase with distance(T, C). Finally, assume that the touch diameter is
  296. equal to the tool thickness, and we arrive at the formulas above.
  297. Finger Tracking
  298. ---------------
  299. The process of finger tracking, i.e., to assign a unique trackingID to each
  300. initiated contact on the surface, is a Euclidian Bipartite Matching
  301. problem. At each event synchronization, the set of actual contacts is
  302. matched to the set of contacts from the previous synchronization. A full
  303. implementation can be found in [#f3]_.
  304. Gestures
  305. --------
  306. In the specific application of creating gesture events, the TOUCH and WIDTH
  307. parameters can be used to, e.g., approximate finger pressure or distinguish
  308. between index finger and thumb. With the addition of the MINOR parameters,
  309. one can also distinguish between a sweeping finger and a pointing finger,
  310. and with ORIENTATION, one can detect twisting of fingers.
  311. Notes
  312. -----
  313. In order to stay compatible with existing applications, the data reported
  314. in a finger packet must not be recognized as single-touch events.
  315. For type A devices, all finger data bypasses input filtering, since
  316. subsequent events of the same type refer to different fingers.
  317. .. [#f1] Also, the difference (TOOL_X - POSITION_X) can be used to model tilt.
  318. .. [#f2] The list can of course be extended.
  319. .. [#f3] The mtdev project: http://bitmath.org/code/mtdev/.
  320. .. [#f4] See the section on event computation.
  321. .. [#f5] See the section on finger tracking.