← Documents Documentation/arch/s390/vfio-ap.rst GitHub 원문 ↗

Linux 6.18.37 · Architecture

Adjunct Processor Facility

IBM Z AP 암호화 queue를 VFIO mediated device로 KVM guest에 독점 할당하는 architecture, sysfs 구성, hotplug와 migration 절차입니다.

Source pathDocumentation/arch/s390/vfio-ap.rst
Source versionLinux v6.18.37
TranslationDUJINLABS 전문 번역 + 해설

요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.

1. 요약·해설

원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.

요약과 해설

vfio-ap.rst:1-1129

`vfio_ap`는 APID와 APQI의 Cartesian product인 APQN을 host 기본 zcrypt driver pool에서 분리해 mediated matrix device에 할당합니다. secure key를 보호하려면 하나의 APQN이 host 또는 guest 하나에만 속해야 하며, guest CRYCB의 APM·AQM·ADM에는 host configuration과 실제 queue binding을 통과한 항목만 반영됩니다.

운영의 핵심은 `/sys/bus/ap/apmask`·`aqmask`로 queue pool을 나눈 뒤 UUID mdev의 assign attribute로 matrix를 구성하는 것입니다. `guest_matrix`, `dyn`, `ap_config` feature는 실제 전달 matrix, 동적 hotplug, 일괄 mask 변경 지원을 표시합니다. AP device 사용 guest의 live migration은 mdev hot unplug와 target 측 재연결이 필요합니다.

2. 영어 원문 전체

번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.

원문 전체 펼치기
1 ===============================
2 Adjunct Processor (AP) facility
3 ===============================
4
5
6 Introduction
7 ============
8 The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
9 of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
10 The AP devices provide cryptographic functions to all CPUs assigned to a
11 linux system running in an IBM Z system LPAR.
12
13 The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
14 is to make AP cards available to KVM guests using the VFIO mediated device
15 framework. This implementation relies considerably on the s390 virtualization
16 facilities which do most of the hard work of providing direct access to AP
17 devices.
18
19 AP Architectural Overview
20 =========================
21 To facilitate the comprehension of the design, let's start with some
22 definitions:
23
24 * AP adapter
25
26 An AP adapter is an IBM Z adapter card that can perform cryptographic
27 functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
28 assigned to the LPAR in which a linux host is running will be available to
29 the linux host. Each adapter is identified by a number from 0 to 255; however,
30 the maximum adapter number is determined by machine model and/or adapter type.
31 When installed, an AP adapter is accessed by AP instructions executed by any
32 CPU.
33
34 The AP adapter cards are assigned to a given LPAR via the system's Activation
35 Profile which can be edited via the HMC. When the linux host system is IPL'd
36 in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and
37 creates a sysfs device for each assigned adapter. For example, if AP adapters
38 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following
39 sysfs device entries::
40
41 /sys/devices/ap/card04
42 /sys/devices/ap/card0a
43
44 Symbolic links to these devices will also be created in the AP bus devices
45 sub-directory::
46
47 /sys/bus/ap/devices/[card04]
48 /sys/bus/ap/devices/[card04]
49
50 * AP domain
51
52 An adapter is partitioned into domains. An adapter can hold up to 256 domains
53 depending upon the adapter type and hardware configuration. A domain is
54 identified by a number from 0 to 255; however, the maximum domain number is
55 determined by machine model and/or adapter type.. A domain can be thought of
56 as a set of hardware registers and memory used for processing AP commands. A
57 domain can be configured with a secure private key used for clear key
58 encryption. A domain is classified in one of two ways depending upon how it
59 may be accessed:
60
61 * Usage domains are domains that are targeted by an AP instruction to
62 process an AP command.
63
64 * Control domains are domains that are changed by an AP command sent to a
65 usage domain; for example, to set the secure private key for the control
66 domain.
67
68 The AP usage and control domains are assigned to a given LPAR via the system's
69 Activation Profile which can be edited via the HMC. When a linux host system
70 is IPL'd in the LPAR, the AP bus module detects the AP usage and control
71 domains assigned to the LPAR. The domain number of each usage domain and
72 adapter number of each AP adapter are combined to create AP queue devices
73 (see AP Queue section below). The domain number of each control domain will be
74 represented in a bitmask and stored in a sysfs file
75 /sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least
76 significant bit, correspond to domains 0-255.
77
78 * AP Queue
79
80 An AP queue is the means by which an AP command is sent to a usage domain
81 inside a specific adapter. An AP queue is identified by a tuple
82 comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
83 APQI corresponds to a given usage domain number within the adapter. This tuple
84 forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
85 instructions include a field containing the APQN to identify the AP queue to
86 which the AP command is to be sent for processing.
87
88 The AP bus will create a sysfs device for each APQN that can be derived from
89 the cross product of the AP adapter and usage domain numbers detected when the
90 AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage
91 domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the
92 following sysfs entries::
93
94 /sys/devices/ap/card04/04.0006
95 /sys/devices/ap/card04/04.0047
96 /sys/devices/ap/card0a/0a.0006
97 /sys/devices/ap/card0a/0a.0047
98
99 The following symbolic links to these devices will be created in the AP bus
100 devices subdirectory::
101
102 /sys/bus/ap/devices/[04.0006]
103 /sys/bus/ap/devices/[04.0047]
104 /sys/bus/ap/devices/[0a.0006]
105 /sys/bus/ap/devices/[0a.0047]
106
107 * AP Instructions:
108
109 There are three AP instructions:
110
111 * NQAP: to enqueue an AP command-request message to a queue
112 * DQAP: to dequeue an AP command-reply message from a queue
113 * PQAP: to administer the queues
114
115 AP instructions identify the domain that is targeted to process the AP
116 command; this must be one of the usage domains. An AP command may modify a
117 domain that is not one of the usage domains, but the modified domain
118 must be one of the control domains.
119
120 AP and SIE
121 ==========
122 Let's now take a look at how AP instructions executed on a guest are interpreted
123 by the hardware.
124
125 A satellite control block called the Crypto Control Block (CRYCB) is attached to
126 our main hardware virtualization control block. The CRYCB contains an AP Control
127 Block (APCB) that has three fields to identify the adapters, usage domains and
128 control domains assigned to the KVM guest:
129
130 * The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
131 to the KVM guest. Each bit in the mask, from left to right, corresponds to
132 an APID from 0-255. If a bit is set, the corresponding adapter is valid for
133 use by the KVM guest.
134
135 * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
136 assigned to the KVM guest. Each bit in the mask, from left to right,
137 corresponds to an AP queue index (APQI) from 0-255. If a bit is set, the
138 corresponding queue is valid for use by the KVM guest.
139
140 * The AP Domain Mask field is a bit mask that identifies the AP control domains
141 assigned to the KVM guest. The ADM bit mask controls which domains can be
142 changed by an AP command-request message sent to a usage domain from the
143 guest. Each bit in the mask, from left to right, corresponds to a domain from
144 0-255. If a bit is set, the corresponding domain can be modified by an AP
145 command-request message sent to a usage domain.
146
147 If you recall from the description of an AP Queue, AP instructions include
148 an APQN to identify the AP queue to which an AP command-request message is to be
149 sent (NQAP and PQAP instructions), or from which a command-reply message is to
150 be received (DQAP instruction). The validity of an APQN is defined by the matrix
151 calculated from the APM and AQM; it is the Cartesian product of all assigned
152 adapter numbers (APM) with all assigned queue indexes (AQM). For example, if
153 adapters 1 and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs
154 (1,5), (1,6), (2,5) and (2,6) will be valid for the guest.
155
156 The APQNs can provide secure key functionality - i.e., a private key is stored
157 on the adapter card for each of its domains - so each APQN must be assigned to
158 at most one guest or to the linux host::
159
160 Example 1: Valid configuration:
161 ------------------------------
162 Guest1: adapters 1,2 domains 5,6
163 Guest2: adapter 1,2 domain 7
164
165 This is valid because both guests have a unique set of APQNs:
166 Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
167 Guest2 has APQNs (1,7), (2,7)
168
169 Example 2: Valid configuration:
170 ------------------------------
171 Guest1: adapters 1,2 domains 5,6
172 Guest2: adapters 3,4 domains 5,6
173
174 This is also valid because both guests have a unique set of APQNs:
175 Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
176 Guest2 has APQNs (3,5), (3,6), (4,5), (4,6)
177
178 Example 3: Invalid configuration:
179 --------------------------------
180 Guest1: adapters 1,2 domains 5,6
181 Guest2: adapter 1 domains 6,7
182
183 This is an invalid configuration because both guests have access to
184 APQN (1,6).
185
186 The Design
187 ==========
188 The design introduces three new objects:
189
190 1. AP matrix device
191 2. VFIO AP device driver (vfio_ap.ko)
192 3. VFIO AP mediated pass-through device
193
194 The VFIO AP device driver
195 -------------------------
196 The VFIO AP (vfio_ap) device driver serves the following purposes:
197
198 1. Provides the interfaces to secure APQNs for exclusive use of KVM guests.
199
200 2. Sets up the VFIO mediated device interfaces to manage a vfio_ap mediated
201 device and creates the sysfs interfaces for assigning adapters, usage
202 domains, and control domains comprising the matrix for a KVM guest.
203
204 3. Configures the APM, AQM and ADM in the APCB contained in the CRYCB referenced
205 by a KVM guest's SIE state description to grant the guest access to a matrix
206 of AP devices
207
208 Reserve APQNs for exclusive use of KVM guests
209 ---------------------------------------------
210 The following block diagram illustrates the mechanism by which APQNs are
211 reserved::
212
213 +------------------+
214 7 remove | |
215 +--------------------> cex4queue driver |
216 | | |
217 | +------------------+
218 |
219 |
220 | +------------------+ +----------------+
221 | 5 register driver | | 3 create | |
222 | +----------------> Device core +----------> matrix device |
223 | | | | | |
224 | | +--------^---------+ +----------------+
225 | | |
226 | | +-------------------+
227 | | +-----------------------------------+ |
228 | | | 4 register AP driver | | 2 register device
229 | | | | |
230 +--------+---+-v---+ +--------+-------+-+
231 | | | |
232 | ap_bus +--------------------- > vfio_ap driver |
233 | | 8 probe | |
234 +--------^---------+ +--^--^------------+
235 6 edit | | |
236 apmask | +-----------------------------+ | 11 mdev create
237 aqmask | | 1 modprobe |
238 +--------+-----+---+ +----------------+-+ +----------------+
239 | | | |10 create| mediated |
240 | admin | | VFIO device core |---------> matrix |
241 | + | | | device |
242 +------+-+---------+ +--------^---------+ +--------^-------+
243 | | | |
244 | | 9 create vfio_ap-passthrough | |
245 | +------------------------------+ |
246 +-------------------------------------------------------------+
247 12 assign adapter/domain/control domain
248
249 The process for reserving an AP queue for use by a KVM guest is:
250
251 1. The administrator loads the vfio_ap device driver
252 2. The vfio-ap driver during its initialization will register a single 'matrix'
253 device with the device core. This will serve as the parent device for
254 all vfio_ap mediated devices used to configure an AP matrix for a guest.
255 3. The /sys/devices/vfio_ap/matrix device is created by the device core
256 4. The vfio_ap device driver will register with the AP bus for AP queue devices
257 of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap
258 driver's probe and remove callback interfaces. Devices older than CEX4 queues
259 are not supported to simplify the implementation by not needlessly
260 complicating the design by supporting older devices that will go out of
261 service in the relatively near future, and for which there are few older
262 systems around on which to test.
263 5. The AP bus registers the vfio_ap device driver with the device core
264 6. The administrator edits the AP adapter and queue masks to reserve AP queues
265 for use by the vfio_ap device driver.
266 7. The AP bus removes the AP queues reserved for the vfio_ap driver from the
267 default zcrypt cex4queue driver.
268 8. The AP bus probes the vfio_ap device driver to bind the queues reserved for
269 it.
270 9. The administrator creates a passthrough type vfio_ap mediated device to be
271 used by a guest
272 10. The administrator assigns the adapters, usage domains and control domains
273 to be exclusively used by a guest.
274
275 Set up the VFIO mediated device interfaces
276 ------------------------------------------
277 The VFIO AP device driver utilizes the common interfaces of the VFIO mediated
278 device core driver to:
279
280 * Register an AP mediated bus driver to add a vfio_ap mediated device to and
281 remove it from a VFIO group.
282 * Create and destroy a vfio_ap mediated device
283 * Add a vfio_ap mediated device to and remove it from the AP mediated bus driver
284 * Add a vfio_ap mediated device to and remove it from an IOMMU group
285
286 The following high-level block diagram shows the main components and interfaces
287 of the VFIO AP mediated device driver::
288
289 +-------------+
290 | |
291 | +---------+ | mdev_register_driver() +--------------+
292 | | Mdev | +<-----------------------+ |
293 | | bus | | | vfio_mdev.ko |
294 | | driver | +----------------------->+ |<-> VFIO user
295 | +---------+ | probe()/remove() +--------------+ APIs
296 | |
297 | MDEV CORE |
298 | MODULE |
299 | mdev.ko |
300 | +---------+ | mdev_register_parent() +--------------+
301 | |Physical | +<-----------------------+ |
302 | | device | | | vfio_ap.ko |<-> matrix
303 | |interface| +----------------------->+ | device
304 | +---------+ | callback +--------------+
305 +-------------+
306
307 During initialization of the vfio_ap module, the matrix device is registered
308 with an 'mdev_parent_ops' structure that provides the sysfs attribute
309 structures, mdev functions and callback interfaces for managing the mediated
310 matrix device.
311
312 * sysfs attribute structures:
313
314 supported_type_groups
315 The VFIO mediated device framework supports creation of user-defined
316 mediated device types. These mediated device types are specified
317 via the 'supported_type_groups' structure when a device is registered
318 with the mediated device framework. The registration process creates the
319 sysfs structures for each mediated device type specified in the
320 'mdev_supported_types' sub-directory of the device being registered. Along
321 with the device type, the sysfs attributes of the mediated device type are
322 provided.
323
324 The VFIO AP device driver will register one mediated device type for
325 passthrough devices:
326
327 /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough
328
329 Only the read-only attributes required by the VFIO mdev framework will
330 be provided::
331
332 ... name
333 ... device_api
334 ... available_instances
335 ... device_api
336
337 Where:
338
339 * name:
340 specifies the name of the mediated device type
341 * device_api:
342 the mediated device type's API
343 * available_instances:
344 the number of vfio_ap mediated passthrough devices
345 that can be created
346 * device_api:
347 specifies the VFIO API
348 mdev_attr_groups
349 This attribute group identifies the user-defined sysfs attributes of the
350 mediated device. When a device is registered with the VFIO mediated device
351 framework, the sysfs attribute files identified in the 'mdev_attr_groups'
352 structure will be created in the vfio_ap mediated device's directory. The
353 sysfs attributes for a vfio_ap mediated device are:
354
355 assign_adapter / unassign_adapter:
356 Write-only attributes for assigning/unassigning an AP adapter to/from the
357 vfio_ap mediated device. To assign/unassign an adapter, the APID of the
358 adapter is echoed into the respective attribute file.
359 assign_domain / unassign_domain:
360 Write-only attributes for assigning/unassigning an AP usage domain to/from
361 the vfio_ap mediated device. To assign/unassign a domain, the domain
362 number of the usage domain is echoed into the respective attribute
363 file.
364 matrix:
365 A read-only file for displaying the APQNs derived from the Cartesian
366 product of the adapter and domain numbers assigned to the vfio_ap mediated
367 device.
368 guest_matrix:
369 A read-only file for displaying the APQNs derived from the Cartesian
370 product of the adapter and domain numbers assigned to the APM and AQM
371 fields respectively of the KVM guest's CRYCB. This may differ from the
372 the APQNs assigned to the vfio_ap mediated device if any APQN does not
373 reference a queue device bound to the vfio_ap device driver (i.e., the
374 queue is not in the host's AP configuration).
375 assign_control_domain / unassign_control_domain:
376 Write-only attributes for assigning/unassigning an AP control domain
377 to/from the vfio_ap mediated device. To assign/unassign a control domain,
378 the ID of the domain to be assigned/unassigned is echoed into the
379 respective attribute file.
380 control_domains:
381 A read-only file for displaying the control domain numbers assigned to the
382 vfio_ap mediated device.
383 ap_config:
384 A read/write file that, when written to, allows all three of the
385 vfio_ap mediated device's ap matrix masks to be replaced in one shot.
386 Three masks are given, one for adapters, one for domains, and one for
387 control domains. If the given state cannot be set then no changes are
388 made to the vfio-ap mediated device.
389
390 The format of the data written to ap_config is as follows:
391 {amask},{dmask},{cmask}\n
392
393 \n is a newline character.
394
395 amask, dmask, and cmask are masks identifying which adapters, domains,
396 and control domains should be assigned to the mediated device.
397
398 The format of a mask is as follows:
399 0xNN..NN
400
401 Where NN..NN is 64 hexadecimal characters representing a 256-bit value.
402 The leftmost (highest order) bit represents adapter/domain 0.
403
404 For an example set of masks that represent your mdev's current
405 configuration, simply cat ap_config.
406
407 Setting an adapter or domain number greater than the maximum allowed for
408 the system will result in an error.
409
410 This attribute is intended to be used by automation. End users would be
411 better served using the respective assign/unassign attributes for
412 adapters, domains, and control domains.
413
414 * functions:
415
416 create:
417 allocates the ap_matrix_mdev structure used by the vfio_ap driver to:
418
419 * Store the reference to the KVM structure for the guest using the mdev
420 * Store the AP matrix configuration for the adapters, domains, and control
421 domains assigned via the corresponding sysfs attributes files
422 * Store the AP matrix configuration for the adapters, domains and control
423 domains available to a guest. A guest may not be provided access to APQNs
424 referencing queue devices that do not exist, or are not bound to the
425 vfio_ap device driver.
426
427 remove:
428 deallocates the vfio_ap mediated device's ap_matrix_mdev structure.
429 This will be allowed only if a running guest is not using the mdev.
430
431 * callback interfaces
432
433 open_device:
434 The vfio_ap driver uses this callback to register a
435 VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the matrix mdev
436 devices. The open_device callback is invoked by userspace to connect the
437 VFIO iommu group for the matrix mdev device to the MDEV bus. Access to the
438 KVM structure used to configure the KVM guest is provided via this callback.
439 The KVM structure, is used to configure the guest's access to the AP matrix
440 defined via the vfio_ap mediated device's sysfs attribute files.
441
442 close_device:
443 unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
444 matrix mdev device and deconfigures the guest's AP matrix.
445
446 ioctl:
447 this callback handles the VFIO_DEVICE_GET_INFO and VFIO_DEVICE_RESET ioctls
448 defined by the vfio framework.
449
450 Configure the guest's AP resources
451 ----------------------------------
452 Configuring the AP resources for a KVM guest will be performed when the
453 VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier
454 function is called when userspace connects to KVM. The guest's AP resources are
455 configured via its APCB by:
456
457 * Setting the bits in the APM corresponding to the APIDs assigned to the
458 vfio_ap mediated device via its 'assign_adapter' interface.
459 * Setting the bits in the AQM corresponding to the domains assigned to the
460 vfio_ap mediated device via its 'assign_domain' interface.
461 * Setting the bits in the ADM corresponding to the domain dIDs assigned to the
462 vfio_ap mediated device via its 'assign_control_domains' interface.
463
464 The linux device model precludes passing a device through to a KVM guest that
465 is not bound to the device driver facilitating its pass-through. Consequently,
466 an APQN that does not reference a queue device bound to the vfio_ap device
467 driver will not be assigned to a KVM guest's matrix. The AP architecture,
468 however, does not provide a means to filter individual APQNs from the guest's
469 matrix, so the adapters, domains and control domains assigned to vfio_ap
470 mediated device via its sysfs 'assign_adapter', 'assign_domain' and
471 'assign_control_domain' interfaces will be filtered before providing the AP
472 configuration to a guest:
473
474 * The APIDs of the adapters, the APQIs of the domains and the domain numbers of
475 the control domains assigned to the matrix mdev that are not also assigned to
476 the host's AP configuration will be filtered.
477
478 * Each APQN derived from the Cartesian product of the APIDs and APQIs assigned
479 to the vfio_ap mdev is examined and if any one of them does not reference a
480 queue device bound to the vfio_ap device driver, the adapter will not be
481 plugged into the guest (i.e., the bit corresponding to its APID will not be
482 set in the APM of the guest's APCB).
483
484 The CPU model features for AP
485 -----------------------------
486 The AP stack relies on the presence of the AP instructions as well as three
487 facilities: The AP Facilities Test (APFT) facility; the AP Query
488 Configuration Information (QCI) facility; and the AP Queue Interruption Control
489 facility. These features/facilities are made available to a KVM guest via the
490 following CPU model features:
491
492 1. ap: Indicates whether the AP instructions are installed on the guest. This
493 feature will be enabled by KVM only if the AP instructions are installed
494 on the host.
495
496 2. apft: Indicates the APFT facility is available on the guest. This facility
497 can be made available to the guest only if it is available on the host (i.e.,
498 facility bit 15 is set).
499
500 3. apqci: Indicates the AP QCI facility is available on the guest. This facility
501 can be made available to the guest only if it is available on the host (i.e.,
502 facility bit 12 is set).
503
504 4. apqi: Indicates AP Queue Interruption Control faclity is available on the
505 guest. This facility can be made available to the guest only if it is
506 available on the host (i.e., facility bit 65 is set).
507
508 Note: If the user chooses to specify a CPU model different than the 'host'
509 model to QEMU, the CPU model features and facilities need to be turned on
510 explicitly; for example::
511
512 /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on,apqi=on
513
514 A guest can be precluded from using AP features/facilities by turning them off
515 explicitly; for example::
516
517 /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off
518
519 Note: If the APFT facility is turned off (apft=off) for the guest, the guest
520 will not see any AP devices. The zcrypt device drivers on the guest that
521 register for type 10 and newer AP devices - i.e., the cex4card and cex4queue
522 device drivers - need the APFT facility to ascertain the facilities installed on
523 a given AP device. If the APFT facility is not installed on the guest, then no
524 adapter or domain devices will get created by the AP bus running on the
525 guest because only type 10 and newer devices can be configured for guest use.
526
527 Example
528 =======
529 Let's now provide an example to illustrate how KVM guests may be given
530 access to AP facilities. For this example, we will show how to configure
531 three guests such that executing the lszcrypt command on the guests would
532 look like this:
533
534 Guest1
535 ------
536 =========== ===== ============
537 CARD.DOMAIN TYPE MODE
538 =========== ===== ============
539 05 CEX5C CCA-Coproc
540 05.0004 CEX5C CCA-Coproc
541 05.00ab CEX5C CCA-Coproc
542 06 CEX5A Accelerator
543 06.0004 CEX5A Accelerator
544 06.00ab CEX5A Accelerator
545 =========== ===== ============
546
547 Guest2
548 ------
549 =========== ===== ============
550 CARD.DOMAIN TYPE MODE
551 =========== ===== ============
552 05 CEX5C CCA-Coproc
553 05.0047 CEX5C CCA-Coproc
554 05.00ff CEX5C CCA-Coproc
555 =========== ===== ============
556
557 Guest3
558 ------
559 =========== ===== ============
560 CARD.DOMAIN TYPE MODE
561 =========== ===== ============
562 06 CEX5A Accelerator
563 06.0047 CEX5A Accelerator
564 06.00ff CEX5A Accelerator
565 =========== ===== ============
566
567 These are the steps:
568
569 1. Install the vfio_ap module on the linux host. The dependency chain for the
570 vfio_ap module is:
571 * iommu
572 * s390
573 * zcrypt
574 * vfio
575 * vfio_mdev
576 * vfio_mdev_device
577 * KVM
578
579 To build the vfio_ap module, the kernel build must be configured with the
580 following Kconfig elements selected:
581 * IOMMU_SUPPORT
582 * S390
583 * AP
584 * VFIO
585 * KVM
586
587 If using make menuconfig select the following to build the vfio_ap module::
588
589 -> Device Drivers
590 -> IOMMU Hardware Support
591 select S390 AP IOMMU Support
592 -> VFIO Non-Privileged userspace driver framework
593 -> Mediated device driver frramework
594 -> VFIO driver for Mediated devices
595 -> I/O subsystem
596 -> VFIO support for AP devices
597
598 2. Secure the AP queues to be used by the three guests so that the host can not
599 access them. To secure them, there are two sysfs files that specify
600 bitmasks marking a subset of the APQN range as usable only by the default AP
601 queue device drivers. All remaining APQNs are available for use by
602 any other device driver. The vfio_ap device driver is currently the only
603 non-default device driver. The location of the sysfs files containing the
604 masks are::
605
606 /sys/bus/ap/apmask
607 /sys/bus/ap/aqmask
608
609 The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
610 (APID). Each bit in the mask, from left to right, corresponds to an APID from
611 0-255. If a bit is set, the APID belongs to the subset of APQNs marked as
612 available only to the default AP queue device drivers.
613
614 The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
615 (APQI). Each bit in the mask, from left to right, corresponds to an APQI from
616 0-255. If a bit is set, the APQI belongs to the subset of APQNs marked as
617 available only to the default AP queue device drivers.
618
619 The Cartesian product of the APIDs corresponding to the bits set in the
620 apmask and the APQIs corresponding to the bits set in the aqmask comprise
621 the subset of APQNs that can be used only by the host default device drivers.
622 All other APQNs are available to the non-default device drivers such as the
623 vfio_ap driver.
624
625 Take, for example, the following masks::
626
627 apmask:
628 0x7d00000000000000000000000000000000000000000000000000000000000000
629
630 aqmask:
631 0x8000000000000000000000000000000000000000000000000000000000000000
632
633 The masks indicate:
634
635 * Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host default
636 device drivers.
637
638 * Domain 0 is available for use by the host default device drivers
639
640 * The subset of APQNs available for use only by the default host device
641 drivers are:
642
643 (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0)
644
645 * All other APQNs are available for use by the non-default device drivers.
646
647 The APQN of each AP queue device assigned to the linux host is checked by the
648 AP bus against the set of APQNs derived from the Cartesian product of APIDs
649 and APQIs marked as available to the default AP queue device drivers. If a
650 match is detected, only the default AP queue device drivers will be probed;
651 otherwise, the vfio_ap device driver will be probed.
652
653 By default, the two masks are set to reserve all APQNs for use by the default
654 AP queue device drivers. There are two ways the default masks can be changed:
655
656 1. The sysfs mask files can be edited by echoing a string into the
657 respective sysfs mask file in one of two formats:
658
659 * An absolute hex string starting with 0x - like "0x12345678" - sets
660 the mask. If the given string is shorter than the mask, it is padded
661 with 0s on the right; for example, specifying a mask value of 0x41 is
662 the same as specifying::
663
664 0x4100000000000000000000000000000000000000000000000000000000000000
665
666 Keep in mind that the mask reads from left to right, so the mask
667 above identifies device numbers 1 and 7 (01000001).
668
669 If the string is longer than the mask, the operation is terminated with
670 an error (EINVAL).
671
672 * Individual bits in the mask can be switched on and off by specifying
673 each bit number to be switched in a comma separated list. Each bit
674 number string must be prepended with a ('+') or minus ('-') to indicate
675 the corresponding bit is to be switched on ('+') or off ('-'). Some
676 valid values are:
677
678 - "+0" switches bit 0 on
679 - "-13" switches bit 13 off
680 - "+0x41" switches bit 65 on
681 - "-0xff" switches bit 255 off
682
683 The following example:
684
685 +0,-6,+0x47,-0xf0
686
687 Switches bits 0 and 71 (0x47) on
688
689 Switches bits 6 and 240 (0xf0) off
690
691 Note that the bits not specified in the list remain as they were before
692 the operation.
693
694 2. The masks can also be changed at boot time via parameters on the kernel
695 command line like this:
696
697 ap.apmask=0xffff ap.aqmask=0x40
698
699 This would create the following masks::
700
701 apmask:
702 0xffff000000000000000000000000000000000000000000000000000000000000
703
704 aqmask:
705 0x4000000000000000000000000000000000000000000000000000000000000000
706
707 Resulting in these two pools::
708
709 default drivers pool: adapter 0-15, domain 1
710 alternate drivers pool: adapter 16-255, domains 0, 2-255
711
712 **Note:**
713 Changing a mask such that one or more APQNs will be taken from a vfio_ap
714 mediated device (see below) will fail with an error (EBUSY). A message
715 is logged to the kernel ring buffer which can be viewed with the 'dmesg'
716 command. The output identifies each APQN flagged as 'in use' and identifies
717 the vfio_ap mediated device to which it is assigned; for example:
718
719 Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804
720 Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4
721
722 Securing the APQNs for our example
723 ----------------------------------
724 To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
725 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding
726 APQNs can be removed from the default masks using either of the following
727 commands::
728
729 echo -5,-6 > /sys/bus/ap/apmask
730
731 echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
732
733 Or the masks can be set as follows::
734
735 echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
736 > apmask
737
738 echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \
739 > aqmask
740
741 This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
742 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
743 sysfs directory for the vfio_ap device driver will now contain symbolic links
744 to the AP queue devices bound to it::
745
746 /sys/bus/ap
747 ... [drivers]
748 ...... [vfio_ap]
749 ......... [05.0004]
750 ......... [05.0047]
751 ......... [05.00ab]
752 ......... [05.00ff]
753 ......... [06.0004]
754 ......... [06.0047]
755 ......... [06.00ab]
756 ......... [06.00ff]
757
758 Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)
759 can be bound to the vfio_ap device driver. The reason for this is to
760 simplify the implementation by not needlessly complicating the design by
761 supporting older devices that will go out of service in the relatively near
762 future and for which there are few older systems on which to test.
763
764 The administrator, therefore, must take care to secure only AP queues that
765 can be bound to the vfio_ap device driver. The device type for a given AP
766 queue device can be read from the parent card's sysfs directory. For example,
767 to see the hardware type of the queue 05.0004:
768
769 cat /sys/bus/ap/devices/card05/hwtype
770
771 The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the
772 vfio_ap device driver.
773
774 3. Create the mediated devices needed to configure the AP matrixes for the
775 three guests and to provide an interface to the vfio_ap driver for
776 use by the guests::
777
778 /sys/devices/vfio_ap/matrix/
779 --- [mdev_supported_types]
780 ------ [vfio_ap-passthrough] (passthrough vfio_ap mediated device type)
781 --------- create
782 --------- [devices]
783
784 To create the mediated devices for the three guests::
785
786 uuidgen > create
787 uuidgen > create
788 uuidgen > create
789
790 or
791
792 echo $uuid1 > create
793 echo $uuid2 > create
794 echo $uuid3 > create
795
796 This will create three mediated devices in the [devices] subdirectory named
797 after the UUID written to the create attribute file. We call them $uuid1,
798 $uuid2 and $uuid3 and this is the sysfs directory structure after creation::
799
800 /sys/devices/vfio_ap/matrix/
801 --- [mdev_supported_types]
802 ------ [vfio_ap-passthrough]
803 --------- [devices]
804 ------------ [$uuid1]
805 --------------- assign_adapter
806 --------------- assign_control_domain
807 --------------- assign_domain
808 --------------- matrix
809 --------------- unassign_adapter
810 --------------- unassign_control_domain
811 --------------- unassign_domain
812
813 ------------ [$uuid2]
814 --------------- assign_adapter
815 --------------- assign_control_domain
816 --------------- assign_domain
817 --------------- matrix
818 --------------- unassign_adapter
819 ----------------unassign_control_domain
820 ----------------unassign_domain
821
822 ------------ [$uuid3]
823 --------------- assign_adapter
824 --------------- assign_control_domain
825 --------------- assign_domain
826 --------------- matrix
827 --------------- unassign_adapter
828 ----------------unassign_control_domain
829 ----------------unassign_domain
830
831 Note *****: The vfio_ap mdevs do not persist across reboots unless the
832 mdevctl tool is used to create and persist them.
833
834 4. The administrator now needs to configure the matrixes for the mediated
835 devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).
836
837 This is how the matrix is configured for Guest1::
838
839 echo 5 > assign_adapter
840 echo 6 > assign_adapter
841 echo 4 > assign_domain
842 echo 0xab > assign_domain
843
844 Control domains can similarly be assigned using the assign_control_domain
845 sysfs file.
846
847 If a mistake is made configuring an adapter, domain or control domain,
848 you can use the unassign_xxx files to unassign the adapter, domain or
849 control domain.
850
851 To display the matrix configuration for Guest1::
852
853 cat matrix
854
855 To display the matrix that is or will be assigned to Guest1::
856
857 cat guest_matrix
858
859 This is how the matrix is configured for Guest2::
860
861 echo 5 > assign_adapter
862 echo 0x47 > assign_domain
863 echo 0xff > assign_domain
864
865 This is how the matrix is configured for Guest3::
866
867 echo 6 > assign_adapter
868 echo 0x47 > assign_domain
869 echo 0xff > assign_domain
870
871 In order to successfully assign an adapter:
872
873 * The adapter number specified must represent a value from 0 up to the
874 maximum adapter number configured for the system. If an adapter number
875 higher than the maximum is specified, the operation will terminate with
876 an error (ENODEV).
877
878 Note: The maximum adapter number can be obtained via the sysfs
879 /sys/bus/ap/ap_max_adapter_id attribute file.
880
881 * Each APQN derived from the Cartesian product of the APID of the adapter
882 being assigned and the APQIs of the domains previously assigned:
883
884 - Must only be available to the vfio_ap device driver as specified in the
885 sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even
886 one APQN is reserved for use by the host device driver, the operation
887 will terminate with an error (EADDRNOTAVAIL).
888
889 - Must NOT be assigned to another vfio_ap mediated device. If even one APQN
890 is assigned to another vfio_ap mediated device, the operation will
891 terminate with an error (EBUSY).
892
893 - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and
894 sys/bus/ap/aqmask attribute files are being edited or the operation may
895 terminate with an error (EBUSY).
896
897 In order to successfully assign a domain:
898
899 * The domain number specified must represent a value from 0 up to the
900 maximum domain number configured for the system. If a domain number
901 higher than the maximum is specified, the operation will terminate with
902 an error (ENODEV).
903
904 Note: The maximum domain number can be obtained via the sysfs
905 /sys/bus/ap/ap_max_domain_id attribute file.
906
907 * Each APQN derived from the Cartesian product of the APQI of the domain
908 being assigned and the APIDs of the adapters previously assigned:
909
910 - Must only be available to the vfio_ap device driver as specified in the
911 sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even
912 one APQN is reserved for use by the host device driver, the operation
913 will terminate with an error (EADDRNOTAVAIL).
914
915 - Must NOT be assigned to another vfio_ap mediated device. If even one APQN
916 is assigned to another vfio_ap mediated device, the operation will
917 terminate with an error (EBUSY).
918
919 - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and
920 sys/bus/ap/aqmask attribute files are being edited or the operation may
921 terminate with an error (EBUSY).
922
923 In order to successfully assign a control domain:
924
925 * The domain number specified must represent a value from 0 up to the maximum
926 domain number configured for the system. If a control domain number higher
927 than the maximum is specified, the operation will terminate with an
928 error (ENODEV).
929
930 5. Start Guest1::
931
932 /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
933 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
934
935 7. Start Guest2::
936
937 /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
938 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
939
940 7. Start Guest3::
941
942 /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
943 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...
944
945 When the guest is shut down, the vfio_ap mediated devices may be removed.
946
947 Using our example again, to remove the vfio_ap mediated device $uuid1::
948
949 /sys/devices/vfio_ap/matrix/
950 --- [mdev_supported_types]
951 ------ [vfio_ap-passthrough]
952 --------- [devices]
953 ------------ [$uuid1]
954 --------------- remove
955
956 ::
957
958 echo 1 > remove
959
960 This will remove all of the matrix mdev device's sysfs structures including
961 the mdev device itself. To recreate and reconfigure the matrix mdev device,
962 all of the steps starting with step 3 will have to be performed again. Note
963 that the remove will fail if a guest using the vfio_ap mdev is still running.
964
965 It is not necessary to remove a vfio_ap mdev, but one may want to
966 remove it if no guest will use it during the remaining lifetime of the linux
967 host. If the vfio_ap mdev is removed, one may want to also reconfigure
968 the pool of adapters and queues reserved for use by the default drivers.
969
970 Hot plug/unplug support:
971 ========================
972 An adapter, domain or control domain may be hot plugged into a running KVM
973 guest by assigning it to the vfio_ap mediated device being used by the guest if
974 the following conditions are met:
975
976 * The adapter, domain or control domain must also be assigned to the host's
977 AP configuration.
978
979 * Each APQN derived from the Cartesian product comprised of the APID of the
980 adapter being assigned and the APQIs of the domains assigned must reference a
981 queue device bound to the vfio_ap device driver.
982
983 * To hot plug a domain, each APQN derived from the Cartesian product
984 comprised of the APQI of the domain being assigned and the APIDs of the
985 adapters assigned must reference a queue device bound to the vfio_ap device
986 driver.
987
988 An adapter, domain or control domain may be hot unplugged from a running KVM
989 guest by unassigning it from the vfio_ap mediated device being used by the
990 guest.
991
992 Over-provisioning of AP queues for a KVM guest:
993 ===============================================
994 Over-provisioning is defined herein as the assignment of adapters or domains to
995 a vfio_ap mediated device that do not reference AP devices in the host's AP
996 configuration. The idea here is that when the adapter or domain becomes
997 available, it will be automatically hot-plugged into the KVM guest using
998 the vfio_ap mediated device to which it is assigned as long as each new APQN
999 resulting from plugging it in references a queue device bound to the vfio_ap
1000 device driver.
1002 Driver Features
1003 ===============
1004 The vfio_ap driver exposes a sysfs file containing supported features.
1005 This exists so third party tools (like Libvirt and mdevctl) can query the
1006 availability of specific features.
1008 The features list can be found here: /sys/bus/matrix/devices/matrix/features
1010 Entries are space delimited. Each entry consists of a combination of
1011 alphanumeric and underscore characters.
1013 Example:
1014 cat /sys/bus/matrix/devices/matrix/features
1015 guest_matrix dyn ap_config
1017 the following features are advertised:
1019 ---------------+---------------------------------------------------------------+
1020 | Flag | Description |
1021 +==============+===============================================================+
1022 | guest_matrix | guest_matrix attribute exists. It reports the matrix of |
1023 | | adapters and domains that are or will be passed through to a |
1024 | | guest when the mdev is attached to it. |
1025 +--------------+---------------------------------------------------------------+
1026 | dyn | Indicates hot plug/unplug of AP adapters, domains and control |
1027 | | domains for a guest to which the mdev is attached. |
1028 +------------+-----------------------------------------------------------------+
1029 | ap_config | ap_config interface for one-shot modifications to mdev config |
1030 +--------------+---------------------------------------------------------------+
1032 Limitations
1033 ===========
1034 Live guest migration is not supported for guests using AP devices without
1035 intervention by a system administrator. Before a KVM guest can be migrated,
1036 the vfio_ap mediated device must be removed. Unfortunately, it can not be
1037 removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while
1038 the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,
1039 its mdev can be hot unplugged from the guest in one of two ways:
1041 1. If the KVM guest was started with libvirt, you can hot unplug the mdev via
1042 the following commands:
1044 virsh detach-device <guestname> <path-to-device-xml>
1046 For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
1047 the guest named 'my-guest':
1049 virsh detach-device my-guest ~/config/my-guest-hostdev.xml
1051 The contents of my-guest-hostdev.xml:
1053 .. code-block:: xml
1055 <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
1056 <source>
1057 <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
1058 </source>
1059 </hostdev>
1062 virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"
1064 For example, to hot unplug the vfio_ap mediated device identified on the
1065 qemu command line with 'id=hostdev0' from the guest named 'my-guest':
1067 .. code-block:: sh
1069 virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"
1071 2. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor
1072 to the guest and using the following qemu monitor command:
1074 (QEMU) device-del id=<device-id>
1076 For example, to hot unplug the vfio_ap mediated device that was specified
1077 on the qemu command line with 'id=hostdev0' when the guest was started:
1079 (QEMU) device-del id=hostdev0
1081 After live migration of the KVM guest completes, an AP configuration can be
1082 restored to the KVM guest by hot plugging a vfio_ap mediated device on the target
1083 system into the guest in one of two ways:
1085 1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated
1086 device into the guest via the following virsh commands:
1088 virsh attach-device <guestname> <path-to-device-xml>
1090 For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
1091 the guest named 'my-guest':
1093 virsh attach-device my-guest ~/config/my-guest-hostdev.xml
1095 The contents of my-guest-hostdev.xml:
1097 .. code-block:: xml
1099 <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
1100 <source>
1101 <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
1102 </source>
1103 </hostdev>
1106 virsh qemu-monitor-command <guest-name> --hmp \
1107 "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
1109 For example, to hot plug the vfio_ap mediated device
1110 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
1111 device-id hostdev0:
1113 virsh qemu-monitor-command my-guest --hmp \
1114 "device_add vfio-ap,\
1115 sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
1116 id=hostdev0"
1118 2. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor
1119 to the guest and using the following qemu monitor command:
1121 (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
1123 For example, to plug the vfio_ap mediated device
1124 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
1125 hostdev0:
1127 (QEMU) device-add "vfio-ap,\
1128 sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
1129 id=hostdev0"

3. 한국어 전문 번역

영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.

Adjunct Processor facility 소개

1-18

Adjunct Processor(AP) facility는 AP instruction 세 개와 1개부터 최대 256개의 PCIe cryptographic adapter card로 구성된 IBM Z 암호화 facility입니다. AP device는 IBM Z system LPAR에서 실행되는 Linux system에 할당된 모든 CPU에 암호화 기능을 제공합니다.

AP adapter card는 AP bus를 통해 노출됩니다. `vfio-ap`의 목적은 VFIO mediated device framework를 사용해 AP card를 KVM guest에 제공하는 것입니다. 이 구현은 AP device 직접 접근의 대부분을 처리하는 s390 virtualization facility에 크게 의존합니다.

AP adapter

19-49

AP adapter는 암호화 기능을 수행하는 IBM Z adapter card입니다. LPAR에는 0개부터 256개까지 adapter를 할당할 수 있으며 Linux host가 실행되는 LPAR에 할당된 adapter는 host에서 사용할 수 있습니다. 각 adapter는 0부터 255 사이의 번호로 식별하지만 최대 번호는 machine model과 adapter type에 따라 정해집니다. 설치된 AP adapter에는 어느 CPU에서든 AP instruction으로 접근합니다.

AP adapter card는 HMC에서 편집하는 system Activation Profile을 통해 LPAR에 할당합니다. LPAR에서 Linux host를 IPL하면 AP bus가 할당된 card를 감지해 adapter마다 sysfs device를 만듭니다. adapter 4와 10(`0x0a`)을 할당한 예의 device entry는 다음과 같습니다.

/sys/devices/ap/card04
/sys/devices/ap/card0a

AP bus의 devices 하위 디렉터리에도 다음 symbolic link를 만듭니다.

/sys/bus/ap/devices/[card04]
/sys/bus/ap/devices/[card04]

AP domain

50-77

adapter는 domain으로 분할됩니다. adapter type과 hardware configuration에 따라 최대 256개 domain을 가질 수 있습니다. domain은 0부터 255 사이의 번호로 식별하지만 최대 번호는 machine model과 adapter type에 따라 정해집니다. domain은 AP command 처리에 쓰는 hardware register와 memory의 집합으로 볼 수 있으며 clear key encryption에 사용할 secure private key를 구성할 수 있습니다.

domain 분류접근 방식
Usage domainAP command를 처리하도록 AP instruction이 직접 대상으로 지정하는 domain입니다.
Control domainusage domain으로 보낸 AP command가 변경하는 domain입니다. 예를 들어 control domain의 secure private key를 설정합니다.

AP usage/control domain도 HMC에서 편집하는 Activation Profile로 LPAR에 할당합니다. Linux host를 IPL하면 AP bus module이 할당된 domain을 감지합니다. 각 usage domain 번호는 각 AP adapter 번호와 결합되어 AP queue device를 만들며, control domain 번호는 `/sys/bus/ap/ap_control_domain_mask`의 bitmask에 저장됩니다. mask의 최상위 bit부터 최하위 bit까지 domain 0-255에 대응합니다.

AP Queue와 APQN

78-106

AP queue는 특정 adapter 안의 usage domain으로 AP command를 보내는 통로입니다. queue는 AP adapter ID(APID)와 AP queue index(APQI)의 tuple로 식별합니다. APQI는 adapter 안의 usage domain 번호에 대응하며, 이 tuple이 AP queue를 고유하게 식별하는 AP Queue Number(APQN)를 이룹니다. AP instruction의 APQN field가 command 처리 대상 queue를 지정합니다.

AP bus module이 적재될 때 감지한 adapter 번호와 usage domain 번호의 Cartesian product에서 만들 수 있는 APQN마다 sysfs device가 생성됩니다. adapter 4와 10, usage domain 6과 71(`0x47`)의 예는 다음과 같습니다.

/sys/devices/ap/card04/04.0006
/sys/devices/ap/card04/04.0047
/sys/devices/ap/card0a/0a.0006
/sys/devices/ap/card0a/0a.0047

AP bus devices 하위 디렉터리에는 다음 symbolic link가 생깁니다.

/sys/bus/ap/devices/[04.0006]
/sys/bus/ap/devices/[04.0047]
/sys/bus/ap/devices/[0a.0006]
/sys/bus/ap/devices/[0a.0047]

AP instruction

107-119
instruction역할
`NQAP`AP command-request message를 queue에 enqueue합니다.
`DQAP`AP command-reply message를 queue에서 dequeue합니다.
`PQAP`queue를 관리합니다.

AP instruction이 command 처리 대상으로 지정하는 domain은 usage domain이어야 합니다. AP command가 usage domain이 아닌 다른 domain을 변경할 수는 있지만, 그 대상은 control domain이어야 합니다.

AP와 SIE의 CRYCB mask

120-146

guest가 실행한 AP instruction을 hardware가 해석할 수 있도록 주 hardware virtualization control block에 Crypto Control Block(CRYCB)이라는 satellite control block을 연결합니다. CRYCB의 AP Control Block(APCB)은 KVM guest에 할당한 adapter, usage domain, control domain을 식별하는 세 field를 가집니다.

APCB field의미
AP Mask(APM)guest에 할당한 AP adapter의 bitmask입니다. 왼쪽부터 APID 0-255에 대응하고 bit가 설정되면 guest가 해당 adapter를 사용할 수 있습니다.
AP Queue Mask(AQM)guest에 할당한 AP usage domain의 bitmask입니다. 왼쪽부터 APQI 0-255에 대응하고 bit가 설정되면 해당 queue가 guest에 유효합니다.
AP Domain Mask(ADM)guest에 할당한 AP control domain의 bitmask입니다. usage domain으로 보낸 AP command-request가 변경할 수 있는 domain을 제어하며 왼쪽부터 domain 0-255에 대응합니다.

APQN matrix와 독점 할당

147-185

`NQAP`와 `PQAP`는 command-request를 보낼 queue를, `DQAP`는 command-reply를 받을 queue를 APQN으로 지정합니다. guest에 유효한 APQN matrix는 APM의 모든 adapter 번호와 AQM의 모든 queue index를 Cartesian product한 것입니다. adapter 1, 2와 usage domain 5, 6이면 `(1,5)`, `(1,6)`, `(2,5)`, `(2,6)`이 유효합니다.

APQN은 adapter card의 domain별 private key를 이용하는 secure key 기능을 제공할 수 있으므로 각 APQN은 guest 하나 또는 Linux host 하나에만 할당해야 합니다.

APQN 할당 유효성 예제
예제Guest1Guest2판정
1adapter 1,2 / domain 5,6adapter 1,2 / domain 7유효: `(1,7)`, `(2,7)`이 Guest1 APQN과 겹치지 않습니다.
2adapter 1,2 / domain 5,6adapter 3,4 / domain 5,6유효: adapter가 달라 모든 APQN이 고유합니다.
3adapter 1,2 / domain 5,6adapter 1 / domain 6,7무효: 두 guest가 APQN `(1,6)`에 접근합니다.

adapter 집합과 domain 집합의 Cartesian product가 다른 guest와 겹치는지 비교합니다.

설계 object와 driver 목적

186-207

설계는 다음 세 object를 도입합니다.

  • AP matrix device
  • VFIO AP device driver(`vfio_ap.ko`)
  • VFIO AP mediated passthrough device

VFIO AP device driver는 다음 역할을 수행합니다.

  • KVM guest가 독점 사용하도록 APQN을 보호하는 interface를 제공합니다.
  • vfio_ap mediated device를 관리하는 VFIO mediated device interface와 guest matrix를 구성할 adapter, usage domain, control domain 할당용 sysfs interface를 만듭니다.
  • KVM guest의 SIE state description이 참조하는 CRYCB 안 APCB의 APM, AQM, ADM을 구성해 guest가 AP device matrix에 접근하도록 합니다.

KVM guest용 APQN 예약 절차

208-274

AP queue를 KVM guest 전용으로 예약하는 구성 요소와 흐름은 다음과 같습니다.

vfio_ap APQN 예약과 mediated device 생성
administrator`modprobe vfio_ap``vfio_ap` driver
`vfio_ap`device core에 matrix device 등록`/sys/devices/vfio_ap/matrix`
`vfio_ap`AP bus에 queue driver 등록device core
administrator`apmask`·`aqmask` 편집AP bus
AP bus기본 `cex4queue`에서 예약 queue 제거`vfio_ap` probe/bind
administratorpassthrough mdev 생성VFIO device core
mediated matrix deviceadapter/domain/control domain 할당KVM guest

원문 block diagram의 등록, probe, queue 재바인딩, mdev 생성과 matrix 할당 순서를 정리했습니다.

  • 1. administrator가 `vfio_ap` device driver를 적재합니다.
  • 2. 초기화 중 driver가 device core에 단일 `matrix` device를 등록합니다. 이 device는 guest AP matrix를 구성하는 모든 vfio_ap mediated device의 parent입니다.
  • 3. device core가 `/sys/devices/vfio_ap/matrix`를 만듭니다.
  • 4. `vfio_ap`는 type 10 이상(CEX4 이상) AP queue device를 위해 probe/remove callback과 함께 AP bus에 등록합니다. 구현 단순성과 시험 가능한 hardware 범위를 고려해 CEX4 이전 queue는 지원하지 않습니다.
  • 5. AP bus가 device core에 `vfio_ap` driver를 등록합니다.
  • 6. administrator가 AP adapter와 queue mask를 편집해 `vfio_ap`용 queue를 예약합니다.
  • 7. AP bus가 예약 queue를 기본 zcrypt `cex4queue` driver에서 제거합니다.
  • 8. AP bus가 `vfio_ap`를 probe해 예약 queue를 bind합니다.
  • 9. administrator가 guest용 passthrough type vfio_ap mediated device를 만듭니다.
  • 10. guest가 독점 사용할 adapter, usage domain, control domain을 할당합니다.

VFIO mediated device interface

275-311

VFIO AP driver는 VFIO mediated device core의 공통 interface로 다음 작업을 수행합니다.

  • AP mediated bus driver를 등록하여 vfio_ap mediated device를 VFIO group에 추가하거나 제거합니다.
  • vfio_ap mediated device를 생성하고 제거합니다.
  • vfio_ap mediated device를 AP mediated bus driver에 추가하거나 제거합니다.
  • vfio_ap mediated device를 IOMMU group에 추가하거나 제거합니다.
VFIO AP mediated device 구성 요소
MDEV core `mdev.ko`Mdev bus driver`mdev_register_driver()``vfio_mdev.ko`
`vfio_mdev.ko``probe()` / `remove()`VFIO user APIs
MDEV core `mdev.ko`physical device interface`mdev_register_parent()``vfio_ap.ko`
`vfio_ap.ko`parent callbackmatrix device

원문 high-level block diagram의 module과 callback 관계입니다.

`vfio_ap` module 초기화 중 matrix device를 `mdev_parent_ops` 구조체와 함께 등록합니다. 이 구조체는 mediated matrix device를 관리할 sysfs attribute 구조, mdev 함수와 callback interface를 제공합니다.

supported_type_groups

312-348

VFIO mediated device framework는 사용자 정의 mediated device type 생성을 지원합니다. device 등록 시 `supported_type_groups`로 type을 지정하면 등록 절차가 해당 device의 `mdev_supported_types` 하위 디렉터리에 각 type과 그 sysfs attribute를 만듭니다.

VFIO AP driver는 passthrough device type 하나를 등록합니다: `/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough`.

read-only attribute의미
`name`mediated device type 이름입니다.
`device_api`mediated device type의 API, 즉 VFIO API를 지정합니다.
`available_instances`생성할 수 있는 vfio_ap mediated passthrough device 수입니다.

mediated device sysfs attribute

349-413

`mdev_attr_groups`는 사용자 정의 sysfs attribute를 식별합니다. mediated framework에 device를 등록하면 이 구조체에 지정된 attribute file이 vfio_ap mediated device 디렉터리에 생성됩니다.

attribute접근과 역할
`assign_adapter` / `unassign_adapter`write-only입니다. 각각 APID를 써서 adapter를 할당하거나 해제합니다.
`assign_domain` / `unassign_domain`write-only입니다. usage domain 번호를 써서 domain을 할당하거나 해제합니다.
`matrix`read-only입니다. mdev에 할당한 adapter와 domain 번호의 Cartesian product에서 나온 APQN을 표시합니다.
`guest_matrix`read-only입니다. guest CRYCB의 APM과 AQM에 실제 할당했거나 할당할 APQN을 표시합니다. queue가 host AP configuration에 없거나 `vfio_ap`에 bind되지 않으면 mdev의 `matrix`와 다를 수 있습니다.
`assign_control_domain` / `unassign_control_domain`write-only입니다. control domain ID를 써서 할당하거나 해제합니다.
`control_domains`read-only입니다. mdev에 할당된 control domain 번호를 표시합니다.
`ap_config`read/write입니다. adapter, domain, control domain의 세 matrix mask를 한 번에 교체합니다. 요청 상태를 설정할 수 없으면 아무 변경도 적용하지 않습니다.

`ap_config`에 쓰는 data 형식과 각 mask 형식은 다음과 같습니다.

The format of the data written to ap_config is as follows:
{amask},{dmask},{cmask}\n

\n is a newline character.

amask, dmask, and cmask are masks identifying which adapters, domains,
and control domains should be assigned to the mediated device.

The format of a mask is as follows:
0xNN..NN

Where NN..NN is 64 hexadecimal characters representing a 256-bit value.

각 mask의 `NN..NN`은 256-bit 값을 나타내는 16진수 64자이며 가장 왼쪽의 최상위 bit가 adapter/domain 0입니다. 현재 mdev configuration에 해당하는 mask 예는 `cat ap_config`로 읽을 수 있습니다. system 최대값보다 큰 adapter나 domain 번호를 설정하면 오류가 납니다.

`ap_config`는 automation 용도입니다. 일반 사용자는 adapter, domain, control domain 각각의 assign/unassign attribute를 사용하는 편이 낫습니다.

mdev 함수와 callback

414-449
interface동작
`create``ap_matrix_mdev`를 할당합니다. mdev를 사용하는 guest의 KVM 구조 reference, sysfs로 지정한 AP matrix configuration, guest에 실제 제공 가능한 AP matrix configuration을 저장합니다. 존재하지 않거나 `vfio_ap`에 bind되지 않은 queue를 가리키는 APQN은 guest에 제공할 수 없습니다.
`remove`mdev의 `ap_matrix_mdev`를 해제합니다. 실행 중인 guest가 mdev를 사용하지 않을 때만 허용됩니다.
`open_device`matrix mdev용 `VFIO_GROUP_NOTIFY_SET_KVM` notifier callback을 등록합니다. userspace가 matrix mdev의 VFIO IOMMU group을 MDEV bus에 연결할 때 호출되며 guest AP matrix를 구성할 KVM 구조 접근을 제공합니다.
`close_device`notifier callback을 등록 해제하고 guest AP matrix를 해제합니다.
`ioctl`VFIO framework의 `VFIO_DEVICE_GET_INFO`와 `VFIO_DEVICE_RESET` ioctl을 처리합니다.

guest AP resource 구성과 filtering

450-483

userspace가 KVM에 연결할 때 `VFIO_GROUP_NOTIFY_SET_KVM` notifier가 호출되어 guest AP resource를 APCB에 구성합니다. `assign_adapter`로 지정한 APID의 APM bit, `assign_domain`으로 지정한 domain의 AQM bit, `assign_control_domains`로 지정한 domain ID의 ADM bit를 설정합니다.

Linux device model은 passthrough를 제공하는 driver에 bind되지 않은 device를 KVM guest에 전달할 수 없게 합니다. 따라서 `vfio_ap`에 bind된 queue device를 참조하지 않는 APQN은 guest matrix에 넣지 않습니다. AP architecture는 matrix에서 개별 APQN만 제거하는 기능이 없으므로 sysfs에서 할당한 adapter, domain, control domain을 guest AP configuration에 제공하기 전에 filtering합니다.

  • matrix mdev에 할당됐지만 host AP configuration에는 없는 adapter APID, domain APQI와 control domain 번호를 걸러냅니다.
  • 할당한 APID와 APQI의 Cartesian product로 만든 모든 APQN을 검사합니다. 하나라도 `vfio_ap`에 bind된 queue device를 가리키지 않으면 해당 adapter를 guest에 plug하지 않으며 APCB APM의 APID bit를 설정하지 않습니다.

AP CPU model feature

484-526

AP stack은 AP instruction과 AP Facilities Test(APFT), AP Query Configuration Information(QCI), AP Queue Interruption Control 세 facility에 의존합니다. KVM guest에는 다음 CPU model feature로 제공합니다.

feature조건과 의미
`ap`guest에 AP instruction이 설치됐는지 나타냅니다. host에 AP instruction이 있을 때만 KVM이 활성화합니다.
`apft`guest의 APFT facility입니다. host facility bit 15가 설정된 경우에만 제공합니다.
`apqci`guest의 AP QCI facility입니다. host facility bit 12가 설정된 경우에만 제공합니다.
`apqi`guest의 AP Queue Interruption Control facility입니다. host facility bit 65가 설정된 경우에만 제공합니다.

QEMU에 `host`와 다른 CPU model을 지정하면 feature와 facility를 명시적으로 켜야 합니다.

/usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on,apqi=on

반대로 다음처럼 명시적으로 꺼 guest의 AP 사용을 막을 수 있습니다.

/usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off

guest에서 `apft=off`이면 AP device가 전혀 보이지 않습니다. type 10 이상 device를 등록하는 `cex4card`와 `cex4queue` driver는 device facility 확인에 APFT가 필요합니다. guest에 APFT가 없으면 guest AP bus는 adapter나 domain device를 만들지 않습니다.

세 guest의 목표 AP matrix

527-566

다음 예는 세 KVM guest에 AP facility를 제공하여 각 guest의 `lszcrypt` 출력이 아래와 같아지도록 구성합니다.

guestcard/domaintypemode
Guest1`05`, `05.0004`, `05.00ab``CEX5C``CCA-Coproc`
Guest1`06`, `06.0004`, `06.00ab``CEX5A``Accelerator`
Guest2`05`, `05.0047`, `05.00ff``CEX5C``CCA-Coproc`
Guest3`06`, `06.0047`, `06.00ff``CEX5A``Accelerator`

vfio_ap module build와 설치

567-597

Linux host에 `vfio_ap` module을 설치합니다. dependency chain은 다음과 같습니다.

  • `iommu`
  • `s390`
  • `zcrypt`
  • `vfio`
  • `vfio_mdev`
  • `vfio_mdev_device`
  • `KVM`

kernel build에서 `IOMMU_SUPPORT`, `S390`, `AP`, `VFIO`, `KVM` Kconfig를 선택해야 합니다. `make menuconfig`에서는 다음 경로를 선택합니다.

-> Device Drivers
   -> IOMMU Hardware Support
      select S390 AP IOMMU Support
   -> VFIO Non-Privileged userspace driver framework
      -> Mediated device driver frramework
         -> VFIO driver for Mediated devices
-> I/O subsystem
   -> VFIO support for AP devices

host 기본 driver용 APQN mask

598-652

세 guest가 사용할 AP queue를 host에서 접근하지 못하도록 보호합니다. `/sys/bus/ap/apmask`와 `/sys/bus/ap/aqmask`의 bitmask는 APQN 범위 중 기본 AP queue driver만 사용할 subset을 표시합니다. 나머지 APQN은 다른 driver가 사용할 수 있으며 현재 non-default driver는 `vfio_ap`뿐입니다.

/sys/bus/ap/apmask
/sys/bus/ap/aqmask

`apmask`는 APID 0-255에 대응하는 256-bit mask이고, `aqmask`는 APQI 0-255에 대응하는 256-bit mask입니다. 왼쪽부터 해당 번호에 대응하며 bit가 설정되면 그 APID/APQI는 기본 AP queue driver 전용 subset에 속합니다.

설정된 `apmask` APID와 `aqmask` APQI의 Cartesian product가 host 기본 driver만 사용할 APQN subset입니다. 나머지는 `vfio_ap` 같은 non-default driver가 사용할 수 있습니다.

다음 mask를 예로 듭니다.

apmask:
0x7d00000000000000000000000000000000000000000000000000000000000000

aqmask:
0x8000000000000000000000000000000000000000000000000000000000000000
  • adapter 1, 2, 3, 4, 5, 7을 host 기본 driver가 사용할 수 있습니다.
  • domain 0을 host 기본 driver가 사용할 수 있습니다.
  • 기본 driver 전용 APQN은 `(1,0)`, `(2,0)`, `(3,0)`, `(4,0)`, `(5,0)`, `(7,0)`입니다.
  • 그 밖의 APQN은 non-default driver가 사용할 수 있습니다.

AP bus는 host에 할당된 각 queue의 APQN을 기본 driver 전용 Cartesian product와 비교합니다. 일치하면 기본 AP queue driver만 probe하고, 일치하지 않으면 `vfio_ap`를 probe합니다.

mask 변경 형식과 충돌

653-721

기본적으로 두 mask는 모든 APQN을 기본 driver용으로 예약합니다. sysfs mask file에 두 형식 중 하나의 문자열을 써서 바꿀 수 있습니다.

`0x12345678`처럼 `0x`로 시작하는 절대 16진수 문자열은 mask 전체를 설정합니다. 문자열이 짧으면 오른쪽을 0으로 채웁니다. 예를 들어 `0x41`은 다음과 같습니다.

0x4100000000000000000000000000000000000000000000000000000000000000

mask를 왼쪽에서 오른쪽으로 읽으므로 이 값은 device 번호 1과 7(`01000001`)을 식별합니다. 문자열이 mask보다 길면 `EINVAL`로 실패합니다.

또는 comma-separated bit 번호 앞에 `+`나 `-`를 붙여 개별 bit를 켜거나 끌 수 있습니다.

값동작
`+0`bit 0을 켭니다.
`-13`bit 13을 끕니다.
`+0x41`bit 65를 켭니다.
`-0xff`bit 255를 끕니다.
+0,-6,+0x47,-0xf0

이 예는 bit 0과 71(`0x47`)을 켜고 bit 6과 240(`0xf0`)을 끕니다. 목록에 없는 bit는 기존 상태를 유지합니다.

kernel command line에서도 mask를 바꿀 수 있습니다.

ap.apmask=0xffff ap.aqmask=0x40

This would create the following masks::

   apmask:
   0xffff000000000000000000000000000000000000000000000000000000000000

   aqmask:
   0x4000000000000000000000000000000000000000000000000000000000000000

Resulting in these two pools::

   default drivers pool:    adapter 0-15, domain 1
   alternate drivers pool:  adapter 16-255, domains 0, 2-255

이 설정은 기본 driver pool을 adapter 0-15와 domain 1로, alternate driver pool을 adapter 16-255와 domain 0 및 2-255로 나눕니다.

mask 변경으로 이미 vfio_ap mediated device에 할당된 APQN을 가져오려 하면 `EBUSY`로 실패합니다. kernel ring buffer의 `dmesg` 메시지는 `in use` APQN과 이를 보유한 mdev UUID를 표시합니다.

Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804
Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4

예제 APQN 보호와 driver bind

722-773

예제의 `05.0004`, `05.0047`, `05.00ab`, `05.00ff`, `06.0004`, `06.0047`, `06.00ab`, `06.00ff` queue를 `vfio_ap`용으로 보호하려면 기본 mask에서 대응 APQN을 다음처럼 제거합니다.

   echo -5,-6 > /sys/bus/ap/apmask

   echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask

Or the masks can be set as follows::

   echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
   > apmask

   echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \
   > aqmask

그러면 여덟 queue가 `vfio_ap` driver에 bind되고 `/sys/bus/ap/drivers/vfio_ap` 아래에 대응 queue symbolic link가 생깁니다.

vfio_ap에 bind된 AP queue
sysfs 위치항목
`/sys/bus/ap/drivers/vfio_ap``05.0004`, `05.0047`, `05.00ab`, `05.00ff`
`/sys/bus/ap/drivers/vfio_ap``06.0004`, `06.0047`, `06.00ab`, `06.00ff`

원문 sysfs tree를 경로와 queue 목록으로 구조화했습니다.

`vfio_ap`에는 type 10 이상 adapter, 즉 CEX4 이상만 bind할 수 있습니다. 이전 device 지원으로 설계를 복잡하게 하지 않고 시험 가능한 hardware에 집중하기 위한 제한입니다.

administrator는 `vfio_ap`에 bind 가능한 queue만 보호해야 합니다. queue `05.0004`의 hardware type은 parent card sysfs에서 다음처럼 읽습니다.

cat /sys/bus/ap/devices/card05/hwtype

bind하려면 `hwtype`이 10 이상, 즉 CEX4 이상이어야 합니다.

mediated device 세 개 생성

774-833

세 guest의 AP matrix를 구성하고 guest가 `vfio_ap` driver를 사용할 interface를 제공하도록 mediated device 세 개를 만듭니다.

vfio_ap-passthrough 생성 경로
`/sys/devices/vfio_ap/matrix``mdev_supported_types``vfio_ap-passthrough`
`create` attributeUUID 쓰기`devices/$uuid`

matrix parent에서 passthrough type의 create attribute와 생성 device 디렉터리로 이어지는 sysfs 계층입니다.

세 device는 다음 두 방식 가운데 하나로 생성합니다.

uuidgen > create
uuidgen > create
uuidgen > create

or

echo $uuid1 > create
echo $uuid2 > create
echo $uuid3 > create

`create`에 쓴 UUID 이름으로 `$uuid1`, `$uuid2`, `$uuid3` 디렉터리가 생기며 각 디렉터리는 다음 attribute를 가집니다.

생성된 matrix mdev sysfs 구조
mdevattribute
`$uuid1``assign_adapter`, `assign_domain`, `assign_control_domain`, `matrix`, `unassign_adapter`, `unassign_domain`, `unassign_control_domain`
`$uuid2`동일한 assign, matrix, unassign attribute
`$uuid3`동일한 assign, matrix, unassign attribute

세 UUID 디렉터리에 동일하게 생성되는 attribute 집합입니다.

`mdevctl`로 생성하고 영속화하지 않으면 vfio_ap mdev는 reboot 뒤 유지되지 않습니다.

세 guest matrix 구성

834-870

Guest1용 `$uuid1`에는 adapter 5, 6과 domain 4, `0xab`를 할당합니다.

echo 5 > assign_adapter
echo 6 > assign_adapter
echo 4 > assign_domain
echo 0xab > assign_domain

control domain도 `assign_control_domain`으로 같은 방식으로 할당합니다. 잘못 할당한 adapter, domain, control domain은 대응 `unassign_xxx` file로 해제합니다. 구성한 matrix와 guest에 실제 할당할 matrix는 각각 다음처럼 확인합니다.

cat matrix
cat guest_matrix

Guest2용 `$uuid2`에는 adapter 5와 domain `0x47`, `0xff`를 할당합니다.

echo 5 > assign_adapter
echo 0x47 > assign_domain
echo 0xff > assign_domain

Guest3용 `$uuid3`에는 adapter 6과 domain `0x47`, `0xff`를 할당합니다.

echo 6 > assign_adapter
echo 0x47 > assign_domain
echo 0xff > assign_domain

adapter·domain·control domain 할당 조건

871-929
대상성공 조건과 오류
adapter번호가 0부터 system 최대 adapter 번호까지여야 하며 초과 시 `ENODEV`입니다. 최대값은 `/sys/bus/ap/ap_max_adapter_id`에서 읽습니다.
adapter에서 파생된 각 APQN`/sys/bus/ap/apmask`와 `/sys/bus/ap/aqmask` 기준으로 `vfio_ap`만 사용할 수 있어야 합니다. host driver 예약 APQN이 하나라도 있으면 `EADDRNOTAVAIL`, 다른 mdev에 할당됐거나 mask 편집 중이면 `EBUSY`가 날 수 있습니다.
domain번호가 0부터 system 최대 domain 번호까지여야 하며 초과 시 `ENODEV`입니다. 최대값은 `/sys/bus/ap/ap_max_domain_id`에서 읽습니다.
domain에서 파생된 각 APQN이전에 할당한 adapter APID와 새 domain APQI의 Cartesian product가 `vfio_ap` 전용이어야 하며, 다른 mdev 할당 또는 mask 편집과 충돌하면 adapter와 같은 오류 규칙을 적용합니다.
control domain번호가 0부터 system 최대 domain 번호까지여야 하며 초과 시 `ENODEV`입니다.

guest 시작과 mdev 제거

930-969

각 guest를 대응 mdev UUID와 AP CPU feature를 지정해 시작합니다.

     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
        -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...

7. Start Guest2::

     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
        -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...

7. Start Guest3::

     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
        -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...

guest 종료 뒤 vfio_ap mediated device를 제거할 수 있습니다. `$uuid1` 디렉터리의 `remove` attribute에 1을 씁니다.

matrix mdev 제거 경로
`/sys/devices/vfio_ap/matrix``mdev_supported_types``vfio_ap-passthrough`
`devices``$uuid1``remove`

원문 sysfs tree의 remove attribute 위치입니다.

echo 1 > remove

이 작업은 mdev 자체를 포함한 모든 matrix mdev sysfs 구조를 제거합니다. 다시 만들려면 3단계부터 반복해야 하며 guest가 mdev를 사용 중이면 제거가 실패합니다.

mdev 제거는 필수가 아니지만 남은 host 수명 동안 어떤 guest도 사용하지 않는다면 제거할 수 있습니다. 제거 뒤에는 기본 driver용 adapter와 queue pool도 다시 구성할 수 있습니다.

hot plug와 hot unplug

970-991

실행 중인 KVM guest가 사용하는 vfio_ap mediated device에 adapter, domain 또는 control domain을 할당하면 다음 조건 아래 hot plug할 수 있습니다.

  • 대상 adapter, domain 또는 control domain이 host AP configuration에도 할당돼 있어야 합니다.
  • adapter를 hot plug할 때 새 adapter APID와 할당된 domain APQI의 Cartesian product에서 나온 모든 APQN이 `vfio_ap`에 bind된 queue device를 가리켜야 합니다.
  • domain을 hot plug할 때 새 domain APQI와 할당된 adapter APID의 Cartesian product에서 나온 모든 APQN이 `vfio_ap`에 bind된 queue device를 가리켜야 합니다.

실행 중인 guest가 사용하는 mdev에서 adapter, domain 또는 control domain을 unassign하면 hot unplug할 수 있습니다.

AP queue over-provisioning

992-1001

여기서 `Over-provisioning`은 host AP configuration에 존재하는 AP device를 참조하지 않는 adapter 또는 domain을 vfio_ap mediated device에 할당하는 것입니다. 나중에 해당 adapter/domain을 사용할 수 있게 되면 새로 생기는 모든 APQN이 `vfio_ap`에 bind된 queue device를 참조하는 한, 이를 할당한 mdev를 사용하는 KVM guest에 자동 hot plug합니다.

driver feature 조회

1002-1031

`vfio_ap` driver는 Libvirt와 `mdevctl` 같은 third-party tool이 특정 feature 가용성을 조회할 수 있도록 `/sys/bus/matrix/devices/matrix/features`를 노출합니다. entry는 공백으로 구분되며 영숫자와 underscore 조합입니다.

cat /sys/bus/matrix/devices/matrix/features
guest_matrix dyn ap_config
flag설명
`guest_matrix``guest_matrix` attribute가 존재하며 mdev가 guest에 attach될 때 전달했거나 전달할 adapter/domain matrix를 보고합니다.
`dyn`mdev가 attach된 guest에서 AP adapter, domain, control domain hot plug/unplug을 지원합니다.
`ap_config`mdev configuration을 한 번에 변경하는 `ap_config` interface를 지원합니다.

live migration 제한과 source guest 분리

1032-1080

AP device를 사용하는 guest의 `Live guest migration`은 system administrator 개입 없이는 지원되지 않습니다. migration 전에 vfio_ap mediated device를 제거해야 하지만 KVM guest가 사용 중이면 `echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove`로 직접 제거할 수 없습니다.

libvirt로 시작한 guest에서는 다음 명령으로 mdev를 hot unplug할 수 있습니다.

virsh detach-device <guestname> <path-to-device-xml>

For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
the guest named 'my-guest':

   virsh detach-device my-guest ~/config/my-guest-hostdev.xml

예제 device XML은 다음과 같습니다.

<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
  <source>
    <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
  </source>
</hostdev>

또는 `virsh qemu-monitor-command`의 HMP `device_del`을 사용합니다.

      virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"

      For example, to hot unplug the vfio_ap mediated device identified on the
      qemu command line with 'id=hostdev0' from the guest named 'my-guest':

.. code-block:: sh

         virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"

QEMU monitor에 직접 연결한 경우 `(QEMU) device-del id=<device-id>`를 사용하며, `id=hostdev0`의 예는 다음과 같습니다.

(QEMU) device-del id=<device-id>

For example, to hot unplug the vfio_ap mediated device that was specified
on the qemu command line with 'id=hostdev0' when the guest was started:

   (QEMU) device-del id=hostdev0

migration 뒤 target guest에 AP 구성 복원

1081-1129

KVM guest live migration 완료 뒤 target system에서 vfio_ap mediated device를 guest에 hot plug해 AP configuration을 복원할 수 있습니다.

libvirt guest에는 `virsh attach-device`와 device XML을 사용합니다.

virsh attach-device <guestname> <path-to-device-xml>

   For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
   the guest named 'my-guest':

      virsh attach-device my-guest ~/config/my-guest-hostdev.xml
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
  <source>
    <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
  </source>
</hostdev>

또는 HMP `device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>`를 `virsh qemu-monitor-command`로 보냅니다.

virsh qemu-monitor-command <guest-name> --hmp \
"device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"

   For example, to hot plug the vfio_ap mediated device
   62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
   device-id hostdev0:

   virsh qemu-monitor-command my-guest --hmp \
   "device_add vfio-ap,\
   sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
   id=hostdev0"

QEMU monitor에 직접 연결한 경우 `(qemu) device_add`를 사용합니다.

(qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"

For example, to plug the vfio_ap mediated device
62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
hostdev0:

   (QEMU) device-add "vfio-ap,\
   sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
   id=hostdev0"