user_mad.rst 5.3 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166
  1. ====================
  2. Userspace MAD access
  3. ====================
  4. Device files
  5. ============
  6. Each port of each InfiniBand device has a "umad" device and an
  7. "issm" device attached. For example, a two-port HCA will have two
  8. umad devices and two issm devices, while a switch will have one
  9. device of each type (for switch port 0).
  10. Creating MAD agents
  11. ===================
  12. A MAD agent can be created by filling in a struct ib_user_mad_reg_req
  13. and then calling the IB_USER_MAD_REGISTER_AGENT ioctl on a file
  14. descriptor for the appropriate device file. If the registration
  15. request succeeds, a 32-bit id will be returned in the structure.
  16. For example::
  17. struct ib_user_mad_reg_req req = { /* ... */ };
  18. ret = ioctl(fd, IB_USER_MAD_REGISTER_AGENT, (char *) &req);
  19. if (!ret)
  20. my_agent = req.id;
  21. else
  22. perror("agent register");
  23. Agents can be unregistered with the IB_USER_MAD_UNREGISTER_AGENT
  24. ioctl. Also, all agents registered through a file descriptor will
  25. be unregistered when the descriptor is closed.
  26. 2014
  27. a new registration ioctl is now provided which allows additional
  28. fields to be provided during registration.
  29. Users of this registration call are implicitly setting the use of
  30. pkey_index (see below).
  31. Receiving MADs
  32. ==============
  33. MADs are received using read(). The receive side now supports
  34. RMPP. The buffer passed to read() must be at least one
  35. struct ib_user_mad + 256 bytes. For example:
  36. If the buffer passed is not large enough to hold the received
  37. MAD (RMPP), the errno is set to ENOSPC and the length of the
  38. buffer needed is set in mad.length.
  39. Example for normal MAD (non RMPP) reads::
  40. struct ib_user_mad *mad;
  41. mad = malloc(sizeof *mad + 256);
  42. ret = read(fd, mad, sizeof *mad + 256);
  43. if (ret != sizeof mad + 256) {
  44. perror("read");
  45. free(mad);
  46. }
  47. Example for RMPP reads::
  48. struct ib_user_mad *mad;
  49. mad = malloc(sizeof *mad + 256);
  50. ret = read(fd, mad, sizeof *mad + 256);
  51. if (ret == -ENOSPC)) {
  52. length = mad.length;
  53. free(mad);
  54. mad = malloc(sizeof *mad + length);
  55. ret = read(fd, mad, sizeof *mad + length);
  56. }
  57. if (ret < 0) {
  58. perror("read");
  59. free(mad);
  60. }
  61. In addition to the actual MAD contents, the other struct ib_user_mad
  62. fields will be filled in with information on the received MAD. For
  63. example, the remote LID will be in mad.lid.
  64. If a send times out, a receive will be generated with mad.status set
  65. to ETIMEDOUT. Otherwise when a MAD has been successfully received,
  66. mad.status will be 0.
  67. poll()/select() may be used to wait until a MAD can be read.
  68. Sending MADs
  69. ============
  70. MADs are sent using write(). The agent ID for sending should be
  71. filled into the id field of the MAD, the destination LID should be
  72. filled into the lid field, and so on. The send side does support
  73. RMPP so arbitrary length MAD can be sent. For example::
  74. struct ib_user_mad *mad;
  75. mad = malloc(sizeof *mad + mad_length);
  76. /* fill in mad->data */
  77. mad->hdr.id = my_agent; /* req.id from agent registration */
  78. mad->hdr.lid = my_dest; /* in network byte order... */
  79. /* etc. */
  80. ret = write(fd, &mad, sizeof *mad + mad_length);
  81. if (ret != sizeof *mad + mad_length)
  82. perror("write");
  83. Transaction IDs
  84. ===============
  85. Users of the umad devices can use the lower 32 bits of the
  86. transaction ID field (that is, the least significant half of the
  87. field in network byte order) in MADs being sent to match
  88. request/response pairs. The upper 32 bits are reserved for use by
  89. the kernel and will be overwritten before a MAD is sent.
  90. P_Key Index Handling
  91. ====================
  92. The old ib_umad interface did not allow setting the P_Key index for
  93. MADs that are sent and did not provide a way for obtaining the P_Key
  94. index of received MADs. A new layout for struct ib_user_mad_hdr
  95. with a pkey_index member has been defined; however, to preserve binary
  96. compatibility with older applications, this new layout will not be used
  97. unless one of IB_USER_MAD_ENABLE_PKEY or IB_USER_MAD_REGISTER_AGENT2 ioctl's
  98. are called before a file descriptor is used for anything else.
  99. In September 2008, the IB_USER_MAD_ABI_VERSION will be incremented
  100. to 6, the new layout of struct ib_user_mad_hdr will be used by
  101. default, and the IB_USER_MAD_ENABLE_PKEY ioctl will be removed.
  102. Setting IsSM Capability Bit
  103. ===========================
  104. To set the IsSM capability bit for a port, simply open the
  105. corresponding issm device file. If the IsSM bit is already set,
  106. then the open call will block until the bit is cleared (or return
  107. immediately with errno set to EAGAIN if the O_NONBLOCK flag is
  108. passed to open()). The IsSM bit will be cleared when the issm file
  109. is closed. No read, write or other operations can be performed on
  110. the issm file.
  111. /dev files
  112. ==========
  113. To create the appropriate character device files automatically with
  114. udev, a rule like::
  115. KERNEL=="umad*", NAME="infiniband/%k"
  116. KERNEL=="issm*", NAME="infiniband/%k"
  117. can be used. This will create device nodes named::
  118. /dev/infiniband/umad0
  119. /dev/infiniband/issm0
  120. for the first port, and so on. The InfiniBand device and port
  121. associated with these devices can be determined from the files::
  122. /sys/class/infiniband_mad/umad0/ibdev
  123. /sys/class/infiniband_mad/umad0/port
  124. and::
  125. /sys/class/infiniband_mad/issm0/ibdev
  126. /sys/class/infiniband_mad/issm0/port