Streaming I/O: Memory Mapping

MMAP streaming 협상과 plane 매핑

입출력 장치가 memory-mapped streaming을 지원하려면 `VIDIOC_QUERYCAP`으로 받은 `v4l2_capability.capabilities`에 `V4L2_CAP_STREAMING`이 있어야 합니다. 두 streaming 방식 가운데 MMAP 지원 여부는 `VIDIOC_REQBUFS`의 memory type을 `V4L2_MEMORY_MMAP`으로 설정해 확인합니다.

Streaming은 응용 프로그램과 드라이버가 buffer pointer만 교환하고 데이터를 복사하지 않는 방식입니다. 원래 장치 메모리, 예를 들어 캡처 확장 그래픽 카드의 video memory를 주소 공간에 매핑하려는 목적이지만 효율이 높아 DMA 가능한 main memory를 할당하는 드라이버도 널리 지원합니다.

드라이버는 고유 buffer type으로 식별되는 여러 독립 buffer set을 지원할 수 있고 set마다 데이터 종류가 다를 수 있습니다. 여러 set을 동시에 쓰려면 logical stream마다 별도 file descriptor를 사용합니다.

.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
.. c:namespace:: V4L

.. _mmap:

******************************
Streaming I/O (Memory Mapping)
******************************

Input and output devices support this I/O method when the
``V4L2_CAP_STREAMING`` flag in the ``capabilities`` field of struct
:c:type:`v4l2_capability` returned by the
:ref:`VIDIOC_QUERYCAP` ioctl is set. There are two
streaming methods, to determine if the memory mapping flavor is
supported applications must call the :ref:`VIDIOC_REQBUFS` ioctl
with the memory type set to ``V4L2_MEMORY_MMAP``.

Streaming is an I/O method where only pointers to buffers are exchanged
between application and driver, the data itself is not copied. Memory
mapping is primarily intended to map buffers in device memory into the
application's address space. Device memory can be for example the video
memory on a graphics card with a video capture add-on. However, being
the most efficient I/O method available for a long time, many other
drivers support streaming as well, allocating buffers in DMA-able main
memory.

A driver can support many sets of buffers. Each set is identified by a
unique buffer type value. The sets are independent and each set can hold
a different type of data. To access different sets at the same time
different file descriptors must be used. [#f1]_

To allocate device buffers applications call the
:ref:`VIDIOC_REQBUFS` ioctl with the desired number
of buffers and buffer type, for example ``V4L2_BUF_TYPE_VIDEO_CAPTURE``.
This ioctl can also be used to change the number of buffers or to free
the allocated memory, provided none of the buffers are still mapped.

Before applications can access the buffers they must map them into their
address space with the :c:func:`mmap()` function. The
location of the buffers in device memory can be determined with the
:ref:`VIDIOC_QUERYBUF` ioctl. In the single-planar
API case, the ``m.offset`` and ``length`` returned in a struct
:c:type:`v4l2_buffer` are passed as sixth and second
parameter to the :c:func:`mmap()` function. When using the
multi-planar API, struct :c:type:`v4l2_buffer` contains an
array of struct :c:type:`v4l2_plane` structures, each
containing its own ``m.offset`` and ``length``. When using the
multi-planar API, every plane of every buffer has to be mapped
separately, so the number of calls to :c:func:`mmap()` should
be equal to number of buffers times number of planes in each buffer. The
offset and length values must not be modified. Remember, the buffers are
allocated in physical memory, as opposed to virtual memory, which can be
swapped out to disk. Applications should free the buffers as soon as
possible with the :c:func:`munmap()` function.

Single-planar 매핑 예제

예제는 capture type과 `V4L2_MEMORY_MMAP`, 요청 count 20으로 `VIDIOC_REQBUFS`를 호출합니다. `EINVAL`이면 video capture 또는 mmap streaming 미지원으로 처리하고, 실제 할당 수가 필요한 최소 5개보다 적으면 중단합니다.

할당된 각 index에 `VIDIOC_QUERYBUF`를 호출해 length와 offset을 얻고 `PROT_READ | PROT_WRITE`, `MAP_SHARED`로 매핑합니다. length는 cleanup의 `munmap()`에 쓰도록 저장합니다. 중간 매핑이 실패했는데 프로세스를 끝내지 않는다면 그때까지 매핑한 buffer를 모두 해제하고 배열도 정리해야 합니다.

Example: Mapping buffers in the single-planar API
=================================================

.. code-block:: c

    struct v4l2_requestbuffers reqbuf;
    struct {
	void *start;
	size_t length;
    } *buffers;
    unsigned int i;

    memset(&reqbuf, 0, sizeof(reqbuf));
    reqbuf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    reqbuf.memory = V4L2_MEMORY_MMAP;
    reqbuf.count = 20;

    if (-1 == ioctl (fd, VIDIOC_REQBUFS, &reqbuf)) {
	if (errno == EINVAL)
	    printf("Video capturing or mmap-streaming is not supported\\n");
	else
	    perror("VIDIOC_REQBUFS");

	exit(EXIT_FAILURE);
    }

    /* We want at least five buffers. */

    if (reqbuf.count < 5) {
	/* You may need to free the buffers here. */
	printf("Not enough buffer memory\\n");
	exit(EXIT_FAILURE);
    }

    buffers = calloc(reqbuf.count, sizeof(*buffers));
    assert(buffers != NULL);

    for (i = 0; i < reqbuf.count; i++) {
	struct v4l2_buffer buffer;

	memset(&buffer, 0, sizeof(buffer));
	buffer.type = reqbuf.type;
	buffer.memory = V4L2_MEMORY_MMAP;
	buffer.index = i;

	if (-1 == ioctl (fd, VIDIOC_QUERYBUF, &buffer)) {
	    perror("VIDIOC_QUERYBUF");
	    exit(EXIT_FAILURE);
	}

	buffers[i].length = buffer.length; /* remember for munmap() */

	buffers[i].start = mmap(NULL, buffer.length,
		    PROT_READ | PROT_WRITE, /* recommended */
		    MAP_SHARED,             /* recommended */
		    fd, buffer.m.offset);

	if (MAP_FAILED == buffers[i].start) {
	    /* If you do not exit here you should unmap() and free()
	       the buffers mapped so far. */
	    perror("mmap");
	    exit(EXIT_FAILURE);
	}
    }

    /* Cleanup. */

    for (i = 0; i < reqbuf.count; i++)
	munmap(buffers[i].start, buffers[i].length);

Multi-planar 매핑 예제

Multi-planar 예제는 현재 형식이 buffer당 3개 plane을 쓴다고 가정하고 `V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE`으로 20개를 요청합니다. 응용 프로그램의 descriptor에는 plane별 start와 length 배열이 필요합니다.

QUERYBUF 전에 `v4l2_plane planes[FMT_NUM_PLANES]`를 준비하고 `v4l2_buffer.length`에 plane 배열 원소 수를, `m.planes`에 배열 포인터를 설정합니다. QUERYBUF 뒤에는 각 plane의 `length`와 `m.mem_offset`을 사용해 따로 `mmap()`합니다.

정리도 buffer와 plane의 이중 loop로 수행합니다. `MAP_FAILED` 뒤 계속 실행한다면 이미 매핑한 모든 buffer와 plane을 역으로 해제해야 합니다.

Example: Mapping buffers in the multi-planar API
================================================

.. code-block:: c

    struct v4l2_requestbuffers reqbuf;
    /* Our current format uses 3 planes per buffer */
    #define FMT_NUM_PLANES = 3

    struct {
	void *start[FMT_NUM_PLANES];
	size_t length[FMT_NUM_PLANES];
    } *buffers;
    unsigned int i, j;

    memset(&reqbuf, 0, sizeof(reqbuf));
    reqbuf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
    reqbuf.memory = V4L2_MEMORY_MMAP;
    reqbuf.count = 20;

    if (ioctl(fd, VIDIOC_REQBUFS, &reqbuf) < 0) {
	if (errno == EINVAL)
	    printf("Video capturing or mmap-streaming is not supported\\n");
	else
	    perror("VIDIOC_REQBUFS");

	exit(EXIT_FAILURE);
    }

    /* We want at least five buffers. */

    if (reqbuf.count < 5) {
	/* You may need to free the buffers here. */
	printf("Not enough buffer memory\\n");
	exit(EXIT_FAILURE);
    }

    buffers = calloc(reqbuf.count, sizeof(*buffers));
    assert(buffers != NULL);

    for (i = 0; i < reqbuf.count; i++) {
	struct v4l2_buffer buffer;
	struct v4l2_plane planes[FMT_NUM_PLANES];

	memset(&buffer, 0, sizeof(buffer));
	buffer.type = reqbuf.type;
	buffer.memory = V4L2_MEMORY_MMAP;
	buffer.index = i;
	/* length in struct v4l2_buffer in multi-planar API stores the size
	 * of planes array. */
	buffer.length = FMT_NUM_PLANES;
	buffer.m.planes = planes;

	if (ioctl(fd, VIDIOC_QUERYBUF, &buffer) < 0) {
	    perror("VIDIOC_QUERYBUF");
	    exit(EXIT_FAILURE);
	}

	/* Every plane has to be mapped separately */
	for (j = 0; j < FMT_NUM_PLANES; j++) {
	    buffers[i].length[j] = buffer.m.planes[j].length; /* remember for munmap() */

	    buffers[i].start[j] = mmap(NULL, buffer.m.planes[j].length,
		     PROT_READ | PROT_WRITE, /* recommended */
		     MAP_SHARED,             /* recommended */
		     fd, buffer.m.planes[j].m.mem_offset);

	    if (MAP_FAILED == buffers[i].start[j]) {
		/* If you do not exit here you should unmap() and free()
		   the buffers and planes mapped so far. */
		perror("mmap");
		exit(EXIT_FAILURE);
	    }
	}
    }

    /* Cleanup. */

    for (i = 0; i < reqbuf.count; i++)
	for (j = 0; j < FMT_NUM_PLANES; j++)
	    munmap(buffers[i].start[j], buffers[i].length[j]);

Incoming/outgoing FIFO와 buffer lifecycle

Streaming 드라이버는 개념적으로 incoming과 outgoing 두 FIFO queue를 유지합니다. video clock에 묶인 동기 capture/output과 디스크·네트워크 지연 및 다른 process의 preemption을 받는 응용 프로그램을 분리해 데이터 손실 가능성을 줄입니다.

출력 buffer는 incoming FIFO에 넣은 순서로 출력되고, 캡처 buffer는 outgoing FIFO에서 꺼내는 순서가 캡처 순서입니다. 드라이버가 항상 queue에 최소 buffer 수를 요구할 수 있지만 그 외에는 미리 넣거나 꺼내 처리할 수에 제한이 없습니다.

응용 프로그램은 dequeued 순서와 다른 순서로 다시 enqueue할 수 있고 드라이버는 enqueue된 빈 buffer를 임의 순서로 채울 수 있습니다. `v4l2_buffer.index`는 buffer 식별자일 뿐 처리 순서를 뜻하지 않습니다.

QBUF/DQBUF로 queue를 조작하고 QUERYBUF로 mapped/enqueued/full/empty 상태를 확인합니다. 기본 DQBUF는 outgoing queue가 비면 block합니다. `open()`에 `O_NONBLOCK`을 썼다면 즉시 `EAGAIN`을 반환하며 `select()`와 `poll()`은 항상 사용할 수 있습니다.

STREAMON/STREAMOFF로 capture 또는 output을 시작·중지합니다. STREAMOFF는 두 queue의 모든 buffer를 제거합니다. 멀티태스킹 환경에서 정확한 '지금'은 없으므로 다른 event와 동기화하려면 capture/output buffer의 `timestamp`를 검사합니다.

Conceptually streaming drivers maintain two buffer queues, an incoming
and an outgoing queue. They separate the synchronous capture or output
operation locked to a video clock from the application which is subject
to random disk or network delays and preemption by other processes,
thereby reducing the probability of data loss. The queues are organized
as FIFOs, buffers will be output in the order enqueued in the incoming
FIFO, and were captured in the order dequeued from the outgoing FIFO.

The driver may require a minimum number of buffers enqueued at all times
to function, apart of this no limit exists on the number of buffers
applications can enqueue in advance, or dequeue and process. They can
also enqueue in a different order than buffers have been dequeued, and
the driver can *fill* enqueued *empty* buffers in any order.  [#f2]_ The
index number of a buffer (struct :c:type:`v4l2_buffer`
``index``) plays no role here, it only identifies the buffer.

Initially all mapped buffers are in dequeued state, inaccessible by the
driver. For capturing applications it is customary to first enqueue all
mapped buffers, then to start capturing and enter the read loop. Here
the application waits until a filled buffer can be dequeued, and
re-enqueues the buffer when the data is no longer needed. Output
applications fill and enqueue buffers, when enough buffers are stacked
up the output is started with :ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>`.
In the write loop, when the application runs out of free buffers, it
must wait until an empty buffer can be dequeued and reused.

To enqueue and dequeue a buffer applications use the
:ref:`VIDIOC_QBUF <VIDIOC_QBUF>` and :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`
ioctl. The status of a buffer being mapped, enqueued, full or empty can
be determined at any time using the :ref:`VIDIOC_QUERYBUF` ioctl. Two
methods exist to suspend execution of the application until one or more
buffers can be dequeued.  By default :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`
blocks when no buffer is in the outgoing queue. When the ``O_NONBLOCK``
flag was given to the :c:func:`open()` function,
:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` returns immediately with an ``EAGAIN``
error code when no buffer is available. The :c:func:`select()`
or :c:func:`poll()` functions are always available.

To start and stop capturing or output applications call the
:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` and :ref:`VIDIOC_STREAMOFF
<VIDIOC_STREAMON>` ioctl.

.. note:::ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>`
   removes all buffers from both queues as a side effect. Since there is
   no notion of doing anything "now" on a multitasking system, if an
   application needs to synchronize with another event it should examine
   the struct ::c:type:`v4l2_buffer` ``timestamp`` of captured
   or outputted buffers.

필수 연산과 설계 주석

한 descriptor에서 QBUF 때마다 buffer type을 바꾸는 것도 이론상 가능하지만 `select()` 의미가 모호해집니다. 오버레이도 CPU를 계속 쓰지 않을 뿐 logical stream이므로 stream마다 descriptor 하나를 쓰는 방식이 명확합니다.

무작위 re-enqueue 순서는 codec처럼 영상을 순서 밖에서 처리하는 응용 프로그램이 buffer를 일찍 반환해 손실을 줄이게 합니다. 무작위 fill 순서는 드라이버가 LIFO로 buffer를 재사용해 cache에 남은 scatter-gather list 등을 활용하게 합니다. 드라이버 수준에서 select와 poll은 같은 연산이며 select는 선택 기능이 아니라 필수 기능입니다.

Drivers implementing memory mapping I/O must support the
:ref:`VIDIOC_REQBUFS <VIDIOC_REQBUFS>`, :ref:`VIDIOC_QUERYBUF
<VIDIOC_QUERYBUF>`, :ref:`VIDIOC_QBUF <VIDIOC_QBUF>`, :ref:`VIDIOC_DQBUF
<VIDIOC_QBUF>`, :ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>`
and :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` ioctls, the :ref:`mmap()
<func-mmap>`, :c:func:`munmap()`, :ref:`select()
<func-select>` and :c:func:`poll()` function. [#f3]_

[capture example]

.. [#f1]
   One could use one file descriptor and set the buffer type field
   accordingly when calling :ref:`VIDIOC_QBUF` etc.,
   but it makes the :c:func:`select()` function ambiguous. We also
   like the clean approach of one file descriptor per logical stream.
   Video overlay for example is also a logical stream, although the CPU
   is not needed for continuous operation.

.. [#f2]
   Random enqueue order permits applications processing images out of
   order (such as video codecs) to return buffers earlier, reducing the
   probability of data loss. Random fill order allows drivers to reuse
   buffers on a LIFO-basis, taking advantage of caches holding
   scatter-gather lists and the like.

.. [#f3]
   At the driver level :c:func:`select()` and :c:func:`poll()` are
   the same, and :c:func:`select()` is too important to be optional.
   The rest should be evident.