user_mad.txt 3.8 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126
  1. USERSPACE MAD ACCESS
  2. Device files
  3. Each port of each InfiniBand device has a "umad" device and an
  4. "issm" device attached. For example, a two-port HCA will have two
  5. umad devices and two issm devices, while a switch will have one
  6. device of each type (for switch port 0).
  7. Creating MAD agents
  8. A MAD agent can be created by filling in a struct ib_user_mad_reg_req
  9. and then calling the IB_USER_MAD_REGISTER_AGENT ioctl on a file
  10. descriptor for the appropriate device file. If the registration
  11. request succeeds, a 32-bit id will be returned in the structure.
  12. For example:
  13. struct ib_user_mad_reg_req req = { /* ... */ };
  14. ret = ioctl(fd, IB_USER_MAD_REGISTER_AGENT, (char *) &req);
  15. if (!ret)
  16. my_agent = req.id;
  17. else
  18. perror("agent register");
  19. Agents can be unregistered with the IB_USER_MAD_UNREGISTER_AGENT
  20. ioctl. Also, all agents registered through a file descriptor will
  21. be unregistered when the descriptor is closed.
  22. Receiving MADs
  23. MADs are received using read(). The receive side now supports
  24. RMPP. The buffer passed to read() must be at least one
  25. struct ib_user_mad + 256 bytes. For example:
  26. If the buffer passed is not large enough to hold the received
  27. MAD (RMPP), the errno is set to ENOSPC and the length of the
  28. buffer needed is set in mad.length.
  29. Example for normal MAD (non RMPP) reads:
  30. struct ib_user_mad *mad;
  31. mad = malloc(sizeof *mad + 256);
  32. ret = read(fd, mad, sizeof *mad + 256);
  33. if (ret != sizeof mad + 256) {
  34. perror("read");
  35. free(mad);
  36. }
  37. Example for RMPP reads:
  38. struct ib_user_mad *mad;
  39. mad = malloc(sizeof *mad + 256);
  40. ret = read(fd, mad, sizeof *mad + 256);
  41. if (ret == -ENOSPC)) {
  42. length = mad.length;
  43. free(mad);
  44. mad = malloc(sizeof *mad + length);
  45. ret = read(fd, mad, sizeof *mad + length);
  46. }
  47. if (ret < 0) {
  48. perror("read");
  49. free(mad);
  50. }
  51. In addition to the actual MAD contents, the other struct ib_user_mad
  52. fields will be filled in with information on the received MAD. For
  53. example, the remote LID will be in mad.lid.
  54. If a send times out, a receive will be generated with mad.status set
  55. to ETIMEDOUT. Otherwise when a MAD has been successfully received,
  56. mad.status will be 0.
  57. poll()/select() may be used to wait until a MAD can be read.
  58. Sending MADs
  59. MADs are sent using write(). The agent ID for sending should be
  60. filled into the id field of the MAD, the destination LID should be
  61. filled into the lid field, and so on. The send side does support
  62. RMPP so arbitrary length MAD can be sent. For example:
  63. struct ib_user_mad *mad;
  64. mad = malloc(sizeof *mad + mad_length);
  65. /* fill in mad->data */
  66. mad->hdr.id = my_agent; /* req.id from agent registration */
  67. mad->hdr.lid = my_dest; /* in network byte order... */
  68. /* etc. */
  69. ret = write(fd, &mad, sizeof *mad + mad_length);
  70. if (ret != sizeof *mad + mad_length)
  71. perror("write");
  72. Setting IsSM Capability Bit
  73. To set the IsSM capability bit for a port, simply open the
  74. corresponding issm device file. If the IsSM bit is already set,
  75. then the open call will block until the bit is cleared (or return
  76. immediately with errno set to EAGAIN if the O_NONBLOCK flag is
  77. passed to open()). The IsSM bit will be cleared when the issm file
  78. is closed. No read, write or other operations can be performed on
  79. the issm file.
  80. /dev files
  81. To create the appropriate character device files automatically with
  82. udev, a rule like
  83. KERNEL="umad*", NAME="infiniband/%k"
  84. KERNEL="issm*", NAME="infiniband/%k"
  85. can be used. This will create device nodes named
  86. /dev/infiniband/umad0
  87. /dev/infiniband/issm0
  88. for the first port, and so on. The InfiniBand device and port
  89. associated with these devices can be determined from the files
  90. /sys/class/infiniband_mad/umad0/ibdev
  91. /sys/class/infiniband_mad/umad0/port
  92. and
  93. /sys/class/infiniband_mad/issm0/ibdev
  94. /sys/class/infiniband_mad/issm0/port