core_locking.txt 4.0 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114
  1. INFINIBAND MIDLAYER LOCKING
  2. This guide is an attempt to make explicit the locking assumptions
  3. made by the InfiniBand midlayer. It describes the requirements on
  4. both low-level drivers that sit below the midlayer and upper level
  5. protocols that use the midlayer.
  6. Sleeping and interrupt context
  7. With the following exceptions, a low-level driver implementation of
  8. all of the methods in struct ib_device may sleep. The exceptions
  9. are any methods from the list:
  10. create_ah
  11. modify_ah
  12. query_ah
  13. destroy_ah
  14. bind_mw
  15. post_send
  16. post_recv
  17. poll_cq
  18. req_notify_cq
  19. map_phys_fmr
  20. which may not sleep and must be callable from any context.
  21. The corresponding functions exported to upper level protocol
  22. consumers:
  23. ib_create_ah
  24. ib_modify_ah
  25. ib_query_ah
  26. ib_destroy_ah
  27. ib_bind_mw
  28. ib_post_send
  29. ib_post_recv
  30. ib_req_notify_cq
  31. ib_map_phys_fmr
  32. are therefore safe to call from any context.
  33. In addition, the function
  34. ib_dispatch_event
  35. used by low-level drivers to dispatch asynchronous events through
  36. the midlayer is also safe to call from any context.
  37. Reentrancy
  38. All of the methods in struct ib_device exported by a low-level
  39. driver must be fully reentrant. The low-level driver is required to
  40. perform all synchronization necessary to maintain consistency, even
  41. if multiple function calls using the same object are run
  42. simultaneously.
  43. The IB midlayer does not perform any serialization of function calls.
  44. Because low-level drivers are reentrant, upper level protocol
  45. consumers are not required to perform any serialization. However,
  46. some serialization may be required to get sensible results. For
  47. example, a consumer may safely call ib_poll_cq() on multiple CPUs
  48. simultaneously. However, the ordering of the work completion
  49. information between different calls of ib_poll_cq() is not defined.
  50. Callbacks
  51. A low-level driver must not perform a callback directly from the
  52. same callchain as an ib_device method call. For example, it is not
  53. allowed for a low-level driver to call a consumer's completion event
  54. handler directly from its post_send method. Instead, the low-level
  55. driver should defer this callback by, for example, scheduling a
  56. tasklet to perform the callback.
  57. The low-level driver is responsible for ensuring that multiple
  58. completion event handlers for the same CQ are not called
  59. simultaneously. The driver must guarantee that only one CQ event
  60. handler for a given CQ is running at a time. In other words, the
  61. following situation is not allowed:
  62. CPU1 CPU2
  63. low-level driver ->
  64. consumer CQ event callback:
  65. /* ... */
  66. ib_req_notify_cq(cq, ...);
  67. low-level driver ->
  68. /* ... */ consumer CQ event callback:
  69. /* ... */
  70. return from CQ event handler
  71. The context in which completion event and asynchronous event
  72. callbacks run is not defined. Depending on the low-level driver, it
  73. may be process context, softirq context, or interrupt context.
  74. Upper level protocol consumers may not sleep in a callback.
  75. Hot-plug
  76. A low-level driver announces that a device is ready for use by
  77. consumers when it calls ib_register_device(), all initialization
  78. must be complete before this call. The device must remain usable
  79. until the driver's call to ib_unregister_device() has returned.
  80. A low-level driver must call ib_register_device() and
  81. ib_unregister_device() from process context. It must not hold any
  82. semaphores that could cause deadlock if a consumer calls back into
  83. the driver across these calls.
  84. An upper level protocol consumer may begin using an IB device as
  85. soon as the add method of its struct ib_client is called for that
  86. device. A consumer must finish all cleanup and free all resources
  87. relating to a device before returning from the remove method.
  88. A consumer is permitted to sleep in its add and remove methods.