I915 GuC Submission/DRM Scheduler RFC

GuC submission·DRM scheduler upstream 계획

Upstream 계획은 basic GuC submission을 먼저 merge하고, 새 parallel submission uAPI를 도입한 뒤 i915를 DRM scheduler로 전환하는 순서입니다.

Basic GuC submission은 모든 gen11+ platform을 지원합니다. 현재 platform에서는 기본 활성화하지 않지만 `enable_guc` module parameter로 켤 수 있습니다. DRM scheduler 통합 과정에서 큰 rework가 예정되어 있으므로 초기 merge는 기능 동작, 중대한 style·layering 오류 부재, execlists regression 방지에 초점을 둡니다.

GuC submission에 맞게 IGT·selftest를 갱신하고 지원 platform의 CI baseline을 활성화한 뒤 CI를 안정화합니다.

기존 bonding uAPI는 GuC submission과 완전히 호환되지 않고 설계 문제도 심각하므로 폐기합니다. 새 uAPI의 `I915_CONTEXT_ENGINES_EXT_PARALLEL` setup은 slot에 N개 context를 구성하고, 한 번의 execbuf IOCTL로 N개 batch를 제출하여 GPU에서 병렬 실행합니다. 처음에는 GuC만 지원하고 필요하면 execlists를 추가합니다.

DRM scheduler 전환에서는 GuC backend의 request queue를 제거하여 backpressure를 scheduler가 처리하게 하고 reset·cancel·watchdog hook을 연결합니다. 그러면 GuC state machine과 locking을 단순화할 수 있습니다.

Execlists backend는 legacy interface이므로 최소한만 DRM scheduler에 연결합니다. Timeslicing·preemption·virtual engine을 완전히 통합하는 비용은 높고, GuC가 대신 처리하므로 ROI가 낮으며 scheduler 복잡성도 커집니다.

i915 priority inheritance·boosting을 DRM scheduler의 선택 기능으로 옮깁니다. 이는 i915 page flip에 쓰이며 다른 DRM driver에도 유용할 수 있습니다. Backend가 preemption·timeslicing을 처리하여 job이 out-of-order로 끝날 수 있으므로 scheduler의 in-order completion 가정을 제거합니다. 마지막으로 i915 priority를 DRM priority level로 바꾸고 필요한 최적화를 수행합니다.

=========================================
I915 GuC Submission/DRM Scheduler Section
=========================================

Upstream plan
=============
For upstream the overall plan for landing GuC submission and integrating the
i915 with the DRM scheduler is:

* Merge basic GuC submission
        * Basic submission support for all gen11+ platforms
        * Not enabled by default on any current platforms but can be enabled via
          modparam enable_guc
        * Lots of rework will need to be done to integrate with DRM scheduler so
          no need to nit pick everything in the code, it just should be
          functional, no major coding style / layering errors, and not regress
          execlists
        * Update IGTs / selftests as needed to work with GuC submission
        * Enable CI on supported platforms for a baseline
        * Rework / get CI heathly for GuC submission in place as needed
* Merge new parallel submission uAPI
        * Bonding uAPI completely incompatible with GuC submission, plus it has
          severe design issues in general, which is why we want to retire it no
          matter what
        * New uAPI adds I915_CONTEXT_ENGINES_EXT_PARALLEL context setup step
          which configures a slot with N contexts
        * After I915_CONTEXT_ENGINES_EXT_PARALLEL a user can submit N batches to
          a slot in a single execbuf IOCTL and the batches run on the GPU in
          parallel
        * Initially only for GuC submission but execlists can be supported if
          needed
* Convert the i915 to use the DRM scheduler
        * GuC submission backend fully integrated with DRM scheduler
                * All request queues removed from backend (e.g. all backpressure
                  handled in DRM scheduler)
                * Resets / cancels hook in DRM scheduler
                * Watchdog hooks into DRM scheduler
                * Lots of complexity of the GuC backend can be pulled out once
                  integrated with DRM scheduler (e.g. state machine gets
                  simpler, locking gets simpler, etc...)
        * Execlists backend will minimum required to hook in the DRM scheduler
                * Legacy interface
                * Features like timeslicing / preemption / virtual engines would
                  be difficult to integrate with the DRM scheduler and these
                  features are not required for GuC submission as the GuC does
                  these things for us
                * ROI low on fully integrating into DRM scheduler
                * Fully integrating would add lots of complexity to DRM
                  scheduler
        * Port i915 priority inheritance / boosting feature in DRM scheduler
                * Used for i915 page flip, may be useful to other DRM drivers as
                  well
                * Will be an optional feature in the DRM scheduler
        * Remove in-order completion assumptions from DRM scheduler
                * Even when using the DRM scheduler the backends will handle
                  preemption, timeslicing, etc... so it is possible for jobs to
                  finish out of order
        * Pull out i915 priority levels and use DRM priority levels
        * Optimize DRM scheduler as needed

GuC TODO와 static priority map

GuC submission upstream의 남은 항목은 error-state capture를 가능하게 하는 GuC firmware·i915 update, GuC log decode용 open-source tool, 공개 GuC specification입니다.

Basic GuC submission에는 큰 uAPI 변경이 필요하지 않습니다. 새 scheduler attribute `I915_SCHEDULER_CAP_STATIC_PRIORITY_MAP`만 추가합니다.

이 attribute는 i915 user priority 약 2,000단계를 GuC의 세 user band에 정적으로 mapping함을 나타냅니다. `-1k..-1`은 Low, `0`은 Medium, `1..1k`는 High입니다.

GuC에는 priority band가 네 개뿐이고 최상위 band는 kernel이 예약합니다. 나머지 세 단계는 DRM scheduler priority level과도 정렬됩니다.

TODOs for GuC submission upstream
=================================

* Need an update to GuC firmware / i915 to enable error state capture
* Open source tool to decode GuC logs
* Public GuC spec

New uAPI for basic GuC submission
=================================
No major changes are required to the uAPI for basic GuC submission. The only
change is a new scheduler attribute: I915_SCHEDULER_CAP_STATIC_PRIORITY_MAP.
This attribute indicates the 2k i915 user priority levels are statically mapped
into 3 levels as follows:

* -1k to -1 Low priority
* 0 Medium priority
* 1 to 1k High priority

This is needed because the GuC only has 4 priority bands. The highest priority
band is reserved with the kernel. This aligns with the DRM scheduler priority
levels too.

Spec references:
----------------
* https://www.khronos.org/registry/EGL/extensions/IMG/EGL_IMG_context_priority.txt
* https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/chap5.html#devsandqueues-priority
* https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priority-t

Parallel submission과 logical engine mapping

기존 bonding uAPI에서는 execbuf 시점의 `I915_SUBMIT_FENCE`가 활성화될 때까지 single-context인지 parallel submit인지 알 수 없습니다. GuC는 N개 context를 미리 명시적으로 등록하고 한 command로 모두 제출해야 하며 bonding처럼 N을 동적으로 바꾸지 못합니다.

또한 legacy bonding uAPI는 혼란스럽고 직관적이지 않으며, `I915_SUBMIT_FENCE`는 설계상 future fence이므로 계속 지원하기 적합하지 않습니다.

새 parallel submission uAPI는 engine logical mapping 공개, context를 병렬용으로 설정하는 `set_parallel` extension, 한 IOCTL에서 N개 BB를 제출하도록 execbuf2 확장이라는 세 부분입니다.

Gen11+ split-frame처럼 BB를 logical order의 engine instance에 배치해야 하는 use case가 있습니다. Fusing에 따라 logical mapping이 바뀌므로 UMD가 fusing을 알게 하지 않고 기존 query engine info IOCTL로 mapping을 공개합니다.

GuC multi-context submission도 engine logical order만 지원합니다. 현재 platform은 engine instance가 최대 두 개이고 logical order가 uAPI order와 같지만 instance가 둘보다 많은 미래 platform에서는 달라집니다.

`drm_i915_engine_info.flags`에 logical instance가 반환되었음을 나타내는 bit를 추가하고, 새 `drm_i915_engine_info.logical_instance` field가 instance 값을 반환합니다.

New parallel submission uAPI
============================
The existing bonding uAPI is completely broken with GuC submission because
whether a submission is a single context submit or parallel submit isn't known
until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple
contexts in parallel with the GuC the context must be explicitly registered with
N contexts and all N contexts must be submitted in a single command to the GuC.
The GuC interfaces do not support dynamically changing between N contexts as the
bonding uAPI does. Hence the need for a new parallel submission interface. Also
the legacy bonding uAPI is quite confusing and not intuitive at all. Furthermore
I915_SUBMIT_FENCE is by design a future fence, so not really something we should
continue to support.

The new parallel submission uAPI consists of 3 parts:

* Export engines logical mapping
* A 'set_parallel' extension to configure contexts for parallel
  submission
* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL

Export engines logical mapping
------------------------------
Certain use cases require BBs to be placed on engine instances in logical order
(e.g. split-frame on gen11+). The logical mapping of engine instances can change
based on fusing. Rather than making UMDs be aware of fusing, simply expose the
logical mapping with the existing query engine info IOCTL. Also the GuC
submission interface currently only supports submitting multiple contexts to
engines in logical order which is a new requirement compared to execlists.
Lastly, all current platforms have at most 2 engine instances and the logical
order is the same as uAPI order. This will change on platforms with more than 2
engine instances.

A single bit will be added to drm_i915_engine_info.flags indicating that the
logical instance has been returned and a new field,
drm_i915_engine_info.logical_instance, returns the logical instance.

set_parallel과 execbuf2 N-BB 제출

`set_parallel` extension은 N개 BB를 병렬 제출할 slot을 구성하는 setup 단계이며, 해당 context를 사용하기 전에 호출해야 합니다. 비슷한 기존 예로 `I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE`와 `I915_CONTEXT_ENGINES_EXT_BOND`가 있습니다.

Slot이 구성되면 한 execbuf2 IOCTL로 N개 BB를 제출합니다. 처음에는 GuC submission만 지원하며 필요하면 나중에 execlists를 추가할 수 있습니다.

uAPI에는 `I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT`과 `drm_i915_context_engines_parallel_submit`을 추가합니다. Kernel-doc은 `include/uapi/drm/i915_drm.h`의 `i915_context_engines_parallel_submit` function을 `rfc` namespace에서 포함합니다.

Parallel slot으로 구성된 context는 한 execbuf2에서 정확히 N개 BB만 제출할 수 있습니다. BB는 `drm_i915_gem_exec_object2` list의 마지막 N개이거나 `I915_EXEC_BATCH_FIRST`가 설정되면 첫 N개입니다.

BB 수는 제출한 slot과 `set_parallel` 또는 다른 extension 설정에서 암시적으로 결정되므로 execbuf2 IOCTL 자체에는 uAPI 변경이 필요 없습니다.

A 'set_parallel' extension to configure contexts for parallel submission
------------------------------------------------------------------------
The 'set_parallel' extension configures a slot for parallel submission of N BBs.
It is a setup step that must be called before using any of the contexts. See
I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for
similar existing examples. Once a slot is configured for parallel submission the
execbuf2 IOCTL can be called submitting N BBs in a single IOCTL. Initially only
supports GuC submission. Execlists supports can be added later if needed.

Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and
drm_i915_context_engines_parallel_submit to the uAPI to implement this
extension.

.. c:namespace-push:: rfc

.. kernel-doc:: include/uapi/drm/i915_drm.h
        :functions: i915_context_engines_parallel_submit

.. c:namespace-pop::

Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL
-------------------------------------------------------------------
Contexts that have been configured with the 'set_parallel' extension can only
submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N objects
in the drm_i915_gem_exec_object2 list or the first N if I915_EXEC_BATCH_FIRST is
set. The number of BBs is implicit based on the slot submitted and how it has
been configured by 'set_parallel' or other extensions. No uAPI changes are
required to the execbuf2 IOCTL.