vfio-ap.rst 35 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866
  1. ===============================
  2. Adjunct Processor (AP) facility
  3. ===============================
  4. Introduction
  5. ============
  6. The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
  7. of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
  8. The AP devices provide cryptographic functions to all CPUs assigned to a
  9. linux system running in an IBM Z system LPAR.
  10. The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
  11. is to make AP cards available to KVM guests using the VFIO mediated device
  12. framework. This implementation relies considerably on the s390 virtualization
  13. facilities which do most of the hard work of providing direct access to AP
  14. devices.
  15. AP Architectural Overview
  16. =========================
  17. To facilitate the comprehension of the design, let's start with some
  18. definitions:
  19. * AP adapter
  20. An AP adapter is an IBM Z adapter card that can perform cryptographic
  21. functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
  22. assigned to the LPAR in which a linux host is running will be available to
  23. the linux host. Each adapter is identified by a number from 0 to 255; however,
  24. the maximum adapter number is determined by machine model and/or adapter type.
  25. When installed, an AP adapter is accessed by AP instructions executed by any
  26. CPU.
  27. The AP adapter cards are assigned to a given LPAR via the system's Activation
  28. Profile which can be edited via the HMC. When the linux host system is IPL'd
  29. in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and
  30. creates a sysfs device for each assigned adapter. For example, if AP adapters
  31. 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following
  32. sysfs device entries::
  33. /sys/devices/ap/card04
  34. /sys/devices/ap/card0a
  35. Symbolic links to these devices will also be created in the AP bus devices
  36. sub-directory::
  37. /sys/bus/ap/devices/[card04]
  38. /sys/bus/ap/devices/[card04]
  39. * AP domain
  40. An adapter is partitioned into domains. An adapter can hold up to 256 domains
  41. depending upon the adapter type and hardware configuration. A domain is
  42. identified by a number from 0 to 255; however, the maximum domain number is
  43. determined by machine model and/or adapter type.. A domain can be thought of
  44. as a set of hardware registers and memory used for processing AP commands. A
  45. domain can be configured with a secure private key used for clear key
  46. encryption. A domain is classified in one of two ways depending upon how it
  47. may be accessed:
  48. * Usage domains are domains that are targeted by an AP instruction to
  49. process an AP command.
  50. * Control domains are domains that are changed by an AP command sent to a
  51. usage domain; for example, to set the secure private key for the control
  52. domain.
  53. The AP usage and control domains are assigned to a given LPAR via the system's
  54. Activation Profile which can be edited via the HMC. When a linux host system
  55. is IPL'd in the LPAR, the AP bus module detects the AP usage and control
  56. domains assigned to the LPAR. The domain number of each usage domain and
  57. adapter number of each AP adapter are combined to create AP queue devices
  58. (see AP Queue section below). The domain number of each control domain will be
  59. represented in a bitmask and stored in a sysfs file
  60. /sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least
  61. significant bit, correspond to domains 0-255.
  62. * AP Queue
  63. An AP queue is the means by which an AP command is sent to a usage domain
  64. inside a specific adapter. An AP queue is identified by a tuple
  65. comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
  66. APQI corresponds to a given usage domain number within the adapter. This tuple
  67. forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
  68. instructions include a field containing the APQN to identify the AP queue to
  69. which the AP command is to be sent for processing.
  70. The AP bus will create a sysfs device for each APQN that can be derived from
  71. the cross product of the AP adapter and usage domain numbers detected when the
  72. AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage
  73. domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the
  74. following sysfs entries::
  75. /sys/devices/ap/card04/04.0006
  76. /sys/devices/ap/card04/04.0047
  77. /sys/devices/ap/card0a/0a.0006
  78. /sys/devices/ap/card0a/0a.0047
  79. The following symbolic links to these devices will be created in the AP bus
  80. devices subdirectory::
  81. /sys/bus/ap/devices/[04.0006]
  82. /sys/bus/ap/devices/[04.0047]
  83. /sys/bus/ap/devices/[0a.0006]
  84. /sys/bus/ap/devices/[0a.0047]
  85. * AP Instructions:
  86. There are three AP instructions:
  87. * NQAP: to enqueue an AP command-request message to a queue
  88. * DQAP: to dequeue an AP command-reply message from a queue
  89. * PQAP: to administer the queues
  90. AP instructions identify the domain that is targeted to process the AP
  91. command; this must be one of the usage domains. An AP command may modify a
  92. domain that is not one of the usage domains, but the modified domain
  93. must be one of the control domains.
  94. AP and SIE
  95. ==========
  96. Let's now take a look at how AP instructions executed on a guest are interpreted
  97. by the hardware.
  98. A satellite control block called the Crypto Control Block (CRYCB) is attached to
  99. our main hardware virtualization control block. The CRYCB contains three fields
  100. to identify the adapters, usage domains and control domains assigned to the KVM
  101. guest:
  102. * The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
  103. to the KVM guest. Each bit in the mask, from left to right (i.e. from most
  104. significant to least significant bit in big endian order), corresponds to
  105. an APID from 0-255. If a bit is set, the corresponding adapter is valid for
  106. use by the KVM guest.
  107. * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
  108. assigned to the KVM guest. Each bit in the mask, from left to right (i.e. from
  109. most significant to least significant bit in big endian order), corresponds to
  110. an AP queue index (APQI) from 0-255. If a bit is set, the corresponding queue
  111. is valid for use by the KVM guest.
  112. * The AP Domain Mask field is a bit mask that identifies the AP control domains
  113. assigned to the KVM guest. The ADM bit mask controls which domains can be
  114. changed by an AP command-request message sent to a usage domain from the
  115. guest. Each bit in the mask, from left to right (i.e. from most significant to
  116. least significant bit in big endian order), corresponds to a domain from
  117. 0-255. If a bit is set, the corresponding domain can be modified by an AP
  118. command-request message sent to a usage domain.
  119. If you recall from the description of an AP Queue, AP instructions include
  120. an APQN to identify the AP queue to which an AP command-request message is to be
  121. sent (NQAP and PQAP instructions), or from which a command-reply message is to
  122. be received (DQAP instruction). The validity of an APQN is defined by the matrix
  123. calculated from the APM and AQM; it is the cross product of all assigned adapter
  124. numbers (APM) with all assigned queue indexes (AQM). For example, if adapters 1
  125. and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs (1,5), (1,6),
  126. (2,5) and (2,6) will be valid for the guest.
  127. The APQNs can provide secure key functionality - i.e., a private key is stored
  128. on the adapter card for each of its domains - so each APQN must be assigned to
  129. at most one guest or to the linux host::
  130. Example 1: Valid configuration:
  131. ------------------------------
  132. Guest1: adapters 1,2 domains 5,6
  133. Guest2: adapter 1,2 domain 7
  134. This is valid because both guests have a unique set of APQNs:
  135. Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
  136. Guest2 has APQNs (1,7), (2,7)
  137. Example 2: Valid configuration:
  138. ------------------------------
  139. Guest1: adapters 1,2 domains 5,6
  140. Guest2: adapters 3,4 domains 5,6
  141. This is also valid because both guests have a unique set of APQNs:
  142. Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
  143. Guest2 has APQNs (3,5), (3,6), (4,5), (4,6)
  144. Example 3: Invalid configuration:
  145. --------------------------------
  146. Guest1: adapters 1,2 domains 5,6
  147. Guest2: adapter 1 domains 6,7
  148. This is an invalid configuration because both guests have access to
  149. APQN (1,6).
  150. The Design
  151. ==========
  152. The design introduces three new objects:
  153. 1. AP matrix device
  154. 2. VFIO AP device driver (vfio_ap.ko)
  155. 3. VFIO AP mediated matrix pass-through device
  156. The VFIO AP device driver
  157. -------------------------
  158. The VFIO AP (vfio_ap) device driver serves the following purposes:
  159. 1. Provides the interfaces to secure APQNs for exclusive use of KVM guests.
  160. 2. Sets up the VFIO mediated device interfaces to manage a mediated matrix
  161. device and creates the sysfs interfaces for assigning adapters, usage
  162. domains, and control domains comprising the matrix for a KVM guest.
  163. 3. Configures the APM, AQM and ADM in the CRYCB referenced by a KVM guest's
  164. SIE state description to grant the guest access to a matrix of AP devices
  165. Reserve APQNs for exclusive use of KVM guests
  166. ---------------------------------------------
  167. The following block diagram illustrates the mechanism by which APQNs are
  168. reserved::
  169. +------------------+
  170. 7 remove | |
  171. +--------------------> cex4queue driver |
  172. | | |
  173. | +------------------+
  174. |
  175. |
  176. | +------------------+ +----------------+
  177. | 5 register driver | | 3 create | |
  178. | +----------------> Device core +----------> matrix device |
  179. | | | | | |
  180. | | +--------^---------+ +----------------+
  181. | | |
  182. | | +-------------------+
  183. | | +-----------------------------------+ |
  184. | | | 4 register AP driver | | 2 register device
  185. | | | | |
  186. +--------+---+-v---+ +--------+-------+-+
  187. | | | |
  188. | ap_bus +--------------------- > vfio_ap driver |
  189. | | 8 probe | |
  190. +--------^---------+ +--^--^------------+
  191. 6 edit | | |
  192. apmask | +-----------------------------+ | 9 mdev create
  193. aqmask | | 1 modprobe |
  194. +--------+-----+---+ +----------------+-+ +----------------+
  195. | | | |8 create | mediated |
  196. | admin | | VFIO device core |---------> matrix |
  197. | + | | | device |
  198. +------+-+---------+ +--------^---------+ +--------^-------+
  199. | | | |
  200. | | 9 create vfio_ap-passthrough | |
  201. | +------------------------------+ |
  202. +-------------------------------------------------------------+
  203. 10 assign adapter/domain/control domain
  204. The process for reserving an AP queue for use by a KVM guest is:
  205. 1. The administrator loads the vfio_ap device driver
  206. 2. The vfio-ap driver during its initialization will register a single 'matrix'
  207. device with the device core. This will serve as the parent device for
  208. all mediated matrix devices used to configure an AP matrix for a guest.
  209. 3. The /sys/devices/vfio_ap/matrix device is created by the device core
  210. 4. The vfio_ap device driver will register with the AP bus for AP queue devices
  211. of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap
  212. driver's probe and remove callback interfaces. Devices older than CEX4 queues
  213. are not supported to simplify the implementation by not needlessly
  214. complicating the design by supporting older devices that will go out of
  215. service in the relatively near future, and for which there are few older
  216. systems around on which to test.
  217. 5. The AP bus registers the vfio_ap device driver with the device core
  218. 6. The administrator edits the AP adapter and queue masks to reserve AP queues
  219. for use by the vfio_ap device driver.
  220. 7. The AP bus removes the AP queues reserved for the vfio_ap driver from the
  221. default zcrypt cex4queue driver.
  222. 8. The AP bus probes the vfio_ap device driver to bind the queues reserved for
  223. it.
  224. 9. The administrator creates a passthrough type mediated matrix device to be
  225. used by a guest
  226. 10. The administrator assigns the adapters, usage domains and control domains
  227. to be exclusively used by a guest.
  228. Set up the VFIO mediated device interfaces
  229. ------------------------------------------
  230. The VFIO AP device driver utilizes the common interface of the VFIO mediated
  231. device core driver to:
  232. * Register an AP mediated bus driver to add a mediated matrix device to and
  233. remove it from a VFIO group.
  234. * Create and destroy a mediated matrix device
  235. * Add a mediated matrix device to and remove it from the AP mediated bus driver
  236. * Add a mediated matrix device to and remove it from an IOMMU group
  237. The following high-level block diagram shows the main components and interfaces
  238. of the VFIO AP mediated matrix device driver::
  239. +-------------+
  240. | |
  241. | +---------+ | mdev_register_driver() +--------------+
  242. | | Mdev | +<-----------------------+ |
  243. | | bus | | | vfio_mdev.ko |
  244. | | driver | +----------------------->+ |<-> VFIO user
  245. | +---------+ | probe()/remove() +--------------+ APIs
  246. | |
  247. | MDEV CORE |
  248. | MODULE |
  249. | mdev.ko |
  250. | +---------+ | mdev_register_device() +--------------+
  251. | |Physical | +<-----------------------+ |
  252. | | device | | | vfio_ap.ko |<-> matrix
  253. | |interface| +----------------------->+ | device
  254. | +---------+ | callback +--------------+
  255. +-------------+
  256. During initialization of the vfio_ap module, the matrix device is registered
  257. with an 'mdev_parent_ops' structure that provides the sysfs attribute
  258. structures, mdev functions and callback interfaces for managing the mediated
  259. matrix device.
  260. * sysfs attribute structures:
  261. supported_type_groups
  262. The VFIO mediated device framework supports creation of user-defined
  263. mediated device types. These mediated device types are specified
  264. via the 'supported_type_groups' structure when a device is registered
  265. with the mediated device framework. The registration process creates the
  266. sysfs structures for each mediated device type specified in the
  267. 'mdev_supported_types' sub-directory of the device being registered. Along
  268. with the device type, the sysfs attributes of the mediated device type are
  269. provided.
  270. The VFIO AP device driver will register one mediated device type for
  271. passthrough devices:
  272. /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough
  273. Only the read-only attributes required by the VFIO mdev framework will
  274. be provided::
  275. ... name
  276. ... device_api
  277. ... available_instances
  278. ... device_api
  279. Where:
  280. * name:
  281. specifies the name of the mediated device type
  282. * device_api:
  283. the mediated device type's API
  284. * available_instances:
  285. the number of mediated matrix passthrough devices
  286. that can be created
  287. * device_api:
  288. specifies the VFIO API
  289. mdev_attr_groups
  290. This attribute group identifies the user-defined sysfs attributes of the
  291. mediated device. When a device is registered with the VFIO mediated device
  292. framework, the sysfs attribute files identified in the 'mdev_attr_groups'
  293. structure will be created in the mediated matrix device's directory. The
  294. sysfs attributes for a mediated matrix device are:
  295. assign_adapter / unassign_adapter:
  296. Write-only attributes for assigning/unassigning an AP adapter to/from the
  297. mediated matrix device. To assign/unassign an adapter, the APID of the
  298. adapter is echoed to the respective attribute file.
  299. assign_domain / unassign_domain:
  300. Write-only attributes for assigning/unassigning an AP usage domain to/from
  301. the mediated matrix device. To assign/unassign a domain, the domain
  302. number of the usage domain is echoed to the respective attribute
  303. file.
  304. matrix:
  305. A read-only file for displaying the APQNs derived from the cross product
  306. of the adapter and domain numbers assigned to the mediated matrix device.
  307. assign_control_domain / unassign_control_domain:
  308. Write-only attributes for assigning/unassigning an AP control domain
  309. to/from the mediated matrix device. To assign/unassign a control domain,
  310. the ID of the domain to be assigned/unassigned is echoed to the respective
  311. attribute file.
  312. control_domains:
  313. A read-only file for displaying the control domain numbers assigned to the
  314. mediated matrix device.
  315. * functions:
  316. create:
  317. allocates the ap_matrix_mdev structure used by the vfio_ap driver to:
  318. * Store the reference to the KVM structure for the guest using the mdev
  319. * Store the AP matrix configuration for the adapters, domains, and control
  320. domains assigned via the corresponding sysfs attributes files
  321. remove:
  322. deallocates the mediated matrix device's ap_matrix_mdev structure. This will
  323. be allowed only if a running guest is not using the mdev.
  324. * callback interfaces
  325. open:
  326. The vfio_ap driver uses this callback to register a
  327. VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix
  328. device. The open is invoked when QEMU connects the VFIO iommu group
  329. for the mdev matrix device to the MDEV bus. Access to the KVM structure used
  330. to configure the KVM guest is provided via this callback. The KVM structure,
  331. is used to configure the guest's access to the AP matrix defined via the
  332. mediated matrix device's sysfs attribute files.
  333. release:
  334. unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
  335. mdev matrix device and deconfigures the guest's AP matrix.
  336. Configure the APM, AQM and ADM in the CRYCB
  337. -------------------------------------------
  338. Configuring the AP matrix for a KVM guest will be performed when the
  339. VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier
  340. function is called when QEMU connects to KVM. The guest's AP matrix is
  341. configured via it's CRYCB by:
  342. * Setting the bits in the APM corresponding to the APIDs assigned to the
  343. mediated matrix device via its 'assign_adapter' interface.
  344. * Setting the bits in the AQM corresponding to the domains assigned to the
  345. mediated matrix device via its 'assign_domain' interface.
  346. * Setting the bits in the ADM corresponding to the domain dIDs assigned to the
  347. mediated matrix device via its 'assign_control_domains' interface.
  348. The CPU model features for AP
  349. -----------------------------
  350. The AP stack relies on the presence of the AP instructions as well as two
  351. facilities: The AP Facilities Test (APFT) facility; and the AP Query
  352. Configuration Information (QCI) facility. These features/facilities are made
  353. available to a KVM guest via the following CPU model features:
  354. 1. ap: Indicates whether the AP instructions are installed on the guest. This
  355. feature will be enabled by KVM only if the AP instructions are installed
  356. on the host.
  357. 2. apft: Indicates the APFT facility is available on the guest. This facility
  358. can be made available to the guest only if it is available on the host (i.e.,
  359. facility bit 15 is set).
  360. 3. apqci: Indicates the AP QCI facility is available on the guest. This facility
  361. can be made available to the guest only if it is available on the host (i.e.,
  362. facility bit 12 is set).
  363. Note: If the user chooses to specify a CPU model different than the 'host'
  364. model to QEMU, the CPU model features and facilities need to be turned on
  365. explicitly; for example::
  366. /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on
  367. A guest can be precluded from using AP features/facilities by turning them off
  368. explicitly; for example::
  369. /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off
  370. Note: If the APFT facility is turned off (apft=off) for the guest, the guest
  371. will not see any AP devices. The zcrypt device drivers that register for type 10
  372. and newer AP devices - i.e., the cex4card and cex4queue device drivers - need
  373. the APFT facility to ascertain the facilities installed on a given AP device. If
  374. the APFT facility is not installed on the guest, then the probe of device
  375. drivers will fail since only type 10 and newer devices can be configured for
  376. guest use.
  377. Example
  378. =======
  379. Let's now provide an example to illustrate how KVM guests may be given
  380. access to AP facilities. For this example, we will show how to configure
  381. three guests such that executing the lszcrypt command on the guests would
  382. look like this:
  383. Guest1
  384. ------
  385. =========== ===== ============
  386. CARD.DOMAIN TYPE MODE
  387. =========== ===== ============
  388. 05 CEX5C CCA-Coproc
  389. 05.0004 CEX5C CCA-Coproc
  390. 05.00ab CEX5C CCA-Coproc
  391. 06 CEX5A Accelerator
  392. 06.0004 CEX5A Accelerator
  393. 06.00ab CEX5C CCA-Coproc
  394. =========== ===== ============
  395. Guest2
  396. ------
  397. =========== ===== ============
  398. CARD.DOMAIN TYPE MODE
  399. =========== ===== ============
  400. 05 CEX5A Accelerator
  401. 05.0047 CEX5A Accelerator
  402. 05.00ff CEX5A Accelerator
  403. =========== ===== ============
  404. Guest3
  405. ------
  406. =========== ===== ============
  407. CARD.DOMAIN TYPE MODE
  408. =========== ===== ============
  409. 06 CEX5A Accelerator
  410. 06.0047 CEX5A Accelerator
  411. 06.00ff CEX5A Accelerator
  412. =========== ===== ============
  413. These are the steps:
  414. 1. Install the vfio_ap module on the linux host. The dependency chain for the
  415. vfio_ap module is:
  416. * iommu
  417. * s390
  418. * zcrypt
  419. * vfio
  420. * vfio_mdev
  421. * vfio_mdev_device
  422. * KVM
  423. To build the vfio_ap module, the kernel build must be configured with the
  424. following Kconfig elements selected:
  425. * IOMMU_SUPPORT
  426. * S390
  427. * ZCRYPT
  428. * S390_AP_IOMMU
  429. * VFIO
  430. * VFIO_MDEV
  431. * VFIO_MDEV_DEVICE
  432. * KVM
  433. If using make menuconfig select the following to build the vfio_ap module::
  434. -> Device Drivers
  435. -> IOMMU Hardware Support
  436. select S390 AP IOMMU Support
  437. -> VFIO Non-Privileged userspace driver framework
  438. -> Mediated device driver frramework
  439. -> VFIO driver for Mediated devices
  440. -> I/O subsystem
  441. -> VFIO support for AP devices
  442. 2. Secure the AP queues to be used by the three guests so that the host can not
  443. access them. To secure them, there are two sysfs files that specify
  444. bitmasks marking a subset of the APQN range as 'usable by the default AP
  445. queue device drivers' or 'not usable by the default device drivers' and thus
  446. available for use by the vfio_ap device driver'. The location of the sysfs
  447. files containing the masks are::
  448. /sys/bus/ap/apmask
  449. /sys/bus/ap/aqmask
  450. The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
  451. (APID). Each bit in the mask, from left to right (i.e., from most significant
  452. to least significant bit in big endian order), corresponds to an APID from
  453. 0-255. If a bit is set, the APID is marked as usable only by the default AP
  454. queue device drivers; otherwise, the APID is usable by the vfio_ap
  455. device driver.
  456. The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
  457. (APQI). Each bit in the mask, from left to right (i.e., from most significant
  458. to least significant bit in big endian order), corresponds to an APQI from
  459. 0-255. If a bit is set, the APQI is marked as usable only by the default AP
  460. queue device drivers; otherwise, the APQI is usable by the vfio_ap device
  461. driver.
  462. Take, for example, the following mask::
  463. 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
  464. It indicates:
  465. 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
  466. belong to the vfio_ap device driver's pool.
  467. The APQN of each AP queue device assigned to the linux host is checked by the
  468. AP bus against the set of APQNs derived from the cross product of APIDs
  469. and APQIs marked as usable only by the default AP queue device drivers. If a
  470. match is detected, only the default AP queue device drivers will be probed;
  471. otherwise, the vfio_ap device driver will be probed.
  472. By default, the two masks are set to reserve all APQNs for use by the default
  473. AP queue device drivers. There are two ways the default masks can be changed:
  474. 1. The sysfs mask files can be edited by echoing a string into the
  475. respective sysfs mask file in one of two formats:
  476. * An absolute hex string starting with 0x - like "0x12345678" - sets
  477. the mask. If the given string is shorter than the mask, it is padded
  478. with 0s on the right; for example, specifying a mask value of 0x41 is
  479. the same as specifying::
  480. 0x4100000000000000000000000000000000000000000000000000000000000000
  481. Keep in mind that the mask reads from left to right (i.e., most
  482. significant to least significant bit in big endian order), so the mask
  483. above identifies device numbers 1 and 7 (01000001).
  484. If the string is longer than the mask, the operation is terminated with
  485. an error (EINVAL).
  486. * Individual bits in the mask can be switched on and off by specifying
  487. each bit number to be switched in a comma separated list. Each bit
  488. number string must be prepended with a ('+') or minus ('-') to indicate
  489. the corresponding bit is to be switched on ('+') or off ('-'). Some
  490. valid values are:
  491. - "+0" switches bit 0 on
  492. - "-13" switches bit 13 off
  493. - "+0x41" switches bit 65 on
  494. - "-0xff" switches bit 255 off
  495. The following example:
  496. +0,-6,+0x47,-0xf0
  497. Switches bits 0 and 71 (0x47) on
  498. Switches bits 6 and 240 (0xf0) off
  499. Note that the bits not specified in the list remain as they were before
  500. the operation.
  501. 2. The masks can also be changed at boot time via parameters on the kernel
  502. command line like this:
  503. ap.apmask=0xffff ap.aqmask=0x40
  504. This would create the following masks::
  505. apmask:
  506. 0xffff000000000000000000000000000000000000000000000000000000000000
  507. aqmask:
  508. 0x4000000000000000000000000000000000000000000000000000000000000000
  509. Resulting in these two pools::
  510. default drivers pool: adapter 0-15, domain 1
  511. alternate drivers pool: adapter 16-255, domains 0, 2-255
  512. Securing the APQNs for our example
  513. ----------------------------------
  514. To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
  515. 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding
  516. APQNs can either be removed from the default masks::
  517. echo -5,-6 > /sys/bus/ap/apmask
  518. echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
  519. Or the masks can be set as follows::
  520. echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
  521. > apmask
  522. echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \
  523. > aqmask
  524. This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
  525. 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
  526. sysfs directory for the vfio_ap device driver will now contain symbolic links
  527. to the AP queue devices bound to it::
  528. /sys/bus/ap
  529. ... [drivers]
  530. ...... [vfio_ap]
  531. ......... [05.0004]
  532. ......... [05.0047]
  533. ......... [05.00ab]
  534. ......... [05.00ff]
  535. ......... [06.0004]
  536. ......... [06.0047]
  537. ......... [06.00ab]
  538. ......... [06.00ff]
  539. Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)
  540. can be bound to the vfio_ap device driver. The reason for this is to
  541. simplify the implementation by not needlessly complicating the design by
  542. supporting older devices that will go out of service in the relatively near
  543. future and for which there are few older systems on which to test.
  544. The administrator, therefore, must take care to secure only AP queues that
  545. can be bound to the vfio_ap device driver. The device type for a given AP
  546. queue device can be read from the parent card's sysfs directory. For example,
  547. to see the hardware type of the queue 05.0004:
  548. cat /sys/bus/ap/devices/card05/hwtype
  549. The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the
  550. vfio_ap device driver.
  551. 3. Create the mediated devices needed to configure the AP matrixes for the
  552. three guests and to provide an interface to the vfio_ap driver for
  553. use by the guests::
  554. /sys/devices/vfio_ap/matrix/
  555. --- [mdev_supported_types]
  556. ------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
  557. --------- create
  558. --------- [devices]
  559. To create the mediated devices for the three guests::
  560. uuidgen > create
  561. uuidgen > create
  562. uuidgen > create
  563. or
  564. echo $uuid1 > create
  565. echo $uuid2 > create
  566. echo $uuid3 > create
  567. This will create three mediated devices in the [devices] subdirectory named
  568. after the UUID written to the create attribute file. We call them $uuid1,
  569. $uuid2 and $uuid3 and this is the sysfs directory structure after creation::
  570. /sys/devices/vfio_ap/matrix/
  571. --- [mdev_supported_types]
  572. ------ [vfio_ap-passthrough]
  573. --------- [devices]
  574. ------------ [$uuid1]
  575. --------------- assign_adapter
  576. --------------- assign_control_domain
  577. --------------- assign_domain
  578. --------------- matrix
  579. --------------- unassign_adapter
  580. --------------- unassign_control_domain
  581. --------------- unassign_domain
  582. ------------ [$uuid2]
  583. --------------- assign_adapter
  584. --------------- assign_control_domain
  585. --------------- assign_domain
  586. --------------- matrix
  587. --------------- unassign_adapter
  588. ----------------unassign_control_domain
  589. ----------------unassign_domain
  590. ------------ [$uuid3]
  591. --------------- assign_adapter
  592. --------------- assign_control_domain
  593. --------------- assign_domain
  594. --------------- matrix
  595. --------------- unassign_adapter
  596. ----------------unassign_control_domain
  597. ----------------unassign_domain
  598. 4. The administrator now needs to configure the matrixes for the mediated
  599. devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).
  600. This is how the matrix is configured for Guest1::
  601. echo 5 > assign_adapter
  602. echo 6 > assign_adapter
  603. echo 4 > assign_domain
  604. echo 0xab > assign_domain
  605. Control domains can similarly be assigned using the assign_control_domain
  606. sysfs file.
  607. If a mistake is made configuring an adapter, domain or control domain,
  608. you can use the unassign_xxx files to unassign the adapter, domain or
  609. control domain.
  610. To display the matrix configuration for Guest1::
  611. cat matrix
  612. This is how the matrix is configured for Guest2::
  613. echo 5 > assign_adapter
  614. echo 0x47 > assign_domain
  615. echo 0xff > assign_domain
  616. This is how the matrix is configured for Guest3::
  617. echo 6 > assign_adapter
  618. echo 0x47 > assign_domain
  619. echo 0xff > assign_domain
  620. In order to successfully assign an adapter:
  621. * The adapter number specified must represent a value from 0 up to the
  622. maximum adapter number configured for the system. If an adapter number
  623. higher than the maximum is specified, the operation will terminate with
  624. an error (ENODEV).
  625. * All APQNs that can be derived from the adapter ID and the IDs of
  626. the previously assigned domains must be bound to the vfio_ap device
  627. driver. If no domains have yet been assigned, then there must be at least
  628. one APQN with the specified APID bound to the vfio_ap driver. If no such
  629. APQNs are bound to the driver, the operation will terminate with an
  630. error (EADDRNOTAVAIL).
  631. No APQN that can be derived from the adapter ID and the IDs of the
  632. previously assigned domains can be assigned to another mediated matrix
  633. device. If an APQN is assigned to another mediated matrix device, the
  634. operation will terminate with an error (EADDRINUSE).
  635. In order to successfully assign a domain:
  636. * The domain number specified must represent a value from 0 up to the
  637. maximum domain number configured for the system. If a domain number
  638. higher than the maximum is specified, the operation will terminate with
  639. an error (ENODEV).
  640. * All APQNs that can be derived from the domain ID and the IDs of
  641. the previously assigned adapters must be bound to the vfio_ap device
  642. driver. If no domains have yet been assigned, then there must be at least
  643. one APQN with the specified APQI bound to the vfio_ap driver. If no such
  644. APQNs are bound to the driver, the operation will terminate with an
  645. error (EADDRNOTAVAIL).
  646. No APQN that can be derived from the domain ID and the IDs of the
  647. previously assigned adapters can be assigned to another mediated matrix
  648. device. If an APQN is assigned to another mediated matrix device, the
  649. operation will terminate with an error (EADDRINUSE).
  650. In order to successfully assign a control domain, the domain number
  651. specified must represent a value from 0 up to the maximum domain number
  652. configured for the system. If a control domain number higher than the maximum
  653. is specified, the operation will terminate with an error (ENODEV).
  654. 5. Start Guest1::
  655. /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
  656. -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
  657. 7. Start Guest2::
  658. /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
  659. -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
  660. 7. Start Guest3::
  661. /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
  662. -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...
  663. When the guest is shut down, the mediated matrix devices may be removed.
  664. Using our example again, to remove the mediated matrix device $uuid1::
  665. /sys/devices/vfio_ap/matrix/
  666. --- [mdev_supported_types]
  667. ------ [vfio_ap-passthrough]
  668. --------- [devices]
  669. ------------ [$uuid1]
  670. --------------- remove
  671. ::
  672. echo 1 > remove
  673. This will remove all of the mdev matrix device's sysfs structures including
  674. the mdev device itself. To recreate and reconfigure the mdev matrix device,
  675. all of the steps starting with step 3 will have to be performed again. Note
  676. that the remove will fail if a guest using the mdev is still running.
  677. It is not necessary to remove an mdev matrix device, but one may want to
  678. remove it if no guest will use it during the remaining lifetime of the linux
  679. host. If the mdev matrix device is removed, one may want to also reconfigure
  680. the pool of adapters and queues reserved for use by the default drivers.
  681. Limitations
  682. ===========
  683. * The KVM/kernel interfaces do not provide a way to prevent restoring an APQN
  684. to the default drivers pool of a queue that is still assigned to a mediated
  685. device in use by a guest. It is incumbent upon the administrator to
  686. ensure there is no mediated device in use by a guest to which the APQN is
  687. assigned lest the host be given access to the private data of the AP queue
  688. device such as a private key configured specifically for the guest.
  689. * Dynamically modifying the AP matrix for a running guest (which would amount to
  690. hot(un)plug of AP devices for the guest) is currently not supported
  691. * Live guest migration is not supported for guests using AP devices.