요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>
===============================================================
Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver
===============================================================
Copyright |copy| 2018 Intel Corporation
Introduction
============
This file documents the Intel IPU3 (3rd generation Image Processing Unit)
Imaging Unit drivers located under drivers/media/pci/intel/ipu3 (CIO2) as well
as under drivers/staging/media/ipu3 (ImgU).
The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
platforms (U/Y processor lines) is made up of two parts namely the Imaging Unit
(ImgU) and the CIO2 device (MIPI CSI2 receiver).
The CIO2 device receives the raw Bayer data from the sensors and outputs the
frames in a format that is specific to the IPU3 (for consumption by the IPU3
ImgU). The CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2*
and is enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
The Imaging Unit (ImgU) is responsible for processing images captured
by the IPU3 CIO2 device. The ImgU driver sources can be found under
drivers/staging/media/ipu3 directory. The driver is enabled through the
CONFIG_VIDEO_IPU3_IMGU config option.
The two driver modules are named ipu3_csi2 and ipu3_imgu, respectively.
The drivers has been tested on Kaby Lake platforms (U/Y processor lines).
Both of the drivers implement V4L2, Media Controller and V4L2 sub-device
interfaces. The IPU3 CIO2 driver supports camera sensors connected to the CIO2
MIPI CSI-2 interfaces through V4L2 sub-device sensor drivers.
CIO2
====
The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
interface to the user space. There is a video node for each CSI-2 receiver,
with a single media controller interface for the entire device.
The CIO2 contains four independent capture channel, each with its own MIPI CSI-2
receiver and DMA engine. Each channel is modelled as a V4L2 sub-device exposed
to userspace as a V4L2 sub-device node and has two pads:
.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
.. flat-table::
:header-rows: 1
* - Pad
- Direction
- Purpose
* - 0
- sink
- MIPI CSI-2 input, connected to the sensor subdev
* - 1
- source
- Raw video capture, connected to the V4L2 video interface
The V4L2 video interfaces model the DMA engines. They are exposed to userspace
as V4L2 video device nodes.
Capturing frames in raw Bayer format
------------------------------------
CIO2 MIPI CSI2 receiver is used to capture frames (in packed raw Bayer format)
from the raw sensors connected to the CSI2 ports. The captured frames are used
as input to the ImgU driver.
Image processing using IPU3 ImgU requires tools such as raw2pnm [#f1]_, and
yavta [#f2]_ due to the following unique requirements and / or features specific
to IPU3.
-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in packed
raw Bayer format that is specific to IPU3.
-- Multiple video nodes have to be operated simultaneously.
Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
2592x1944 image capture.
Using the media controller APIs, the ov5670 sensor is configured to send
frames in packed raw Bayer format to IPU3 CSI2 receiver.
.. code-block:: none
# This example assumes /dev/media0 as the CIO2 media device
export MDEV=/dev/media0
# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036")
# Establish the link for the media devices using media-ctl
media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]"
# Set the format for the media devices
media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]"
media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]"
media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]"
Once the media pipeline is configured, desired sensor specific settings
(such as exposure and gain settings) can be set, using the yavta tool.
e.g
.. code-block:: none
yavta -w 0x009e0903 444 $SDEV
yavta -w 0x009e0913 1024 $SDEV
yavta -w 0x009e0911 2046 $SDEV
Once the desired sensor settings are set, frame captures can be done as below.
e.g
.. code-block:: none
yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \
-f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0")
With the above command, 10 frames are captured at 2592x1944 resolution, with
sGRBG10 format and output as IPU3_SGRBG10 format.
The captured frames are available as /tmp/frame-#.bin files.
ImgU
====
The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
subdev interface to the user space.
Each V4L2 subdev represents a pipe, which can support a maximum of 2 streams.
This helps to support advanced camera features like Continuous View Finder (CVF)
and Snapshot During Video(SDV).
The ImgU contains two independent pipes, each modelled as a V4L2 sub-device
exposed to userspace as a V4L2 sub-device node.
Each pipe has two sink pads and three source pads for the following purpose:
.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
.. flat-table::
:header-rows: 1
* - Pad
- Direction
- Purpose
* - 0
- sink
- Input raw video stream
* - 1
- sink
- Processing parameters
* - 2
- source
- Output processed video stream
* - 3
- source
- Output viewfinder video stream
* - 4
- source
- 3A statistics
Each pad is connected to a corresponding V4L2 video interface, exposed to
userspace as a V4L2 video device node.
Device operation
----------------
With ImgU, once the input video node ("ipu3-imgu 0/1":0, in
<entity>:<pad-number> format) is queued with buffer (in packed raw Bayer
format), ImgU starts processing the buffer and produces the video output in YUV
format and statistics output on respective output nodes. The driver is expected
to have buffers ready for all of parameter, output and statistics nodes, when
input video node is queued with buffer.
At a minimum, all of input, main output, 3A statistics and viewfinder
video nodes should be enabled for IPU3 to start image processing.
Each ImgU V4L2 subdev has the following set of video nodes.
input, output and viewfinder video nodes
----------------------------------------
The frames (in packed raw Bayer format specific to the IPU3) received by the
input video node is processed by the IPU3 Imaging Unit and are output to 2 video
nodes, with each targeting a different purpose (main output and viewfinder
output).
Details onand the Bayer format specific to the IPU3 can be found in
:ref:`v4l2-pix-fmt-ipu3-sbggr10`.
The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
Only the multi-planar API is supported. More details can be found at
:ref:`planar-apis`.
Parameters video node
---------------------
The parameters video node receives the ImgU algorithm parameters that are used
to configure how the ImgU algorithms process the image.
Details on processing parameters specific to the IPU3 can be found in
:ref:`v4l2-meta-fmt-params`.
3A statistics video node
------------------------
3A statistics video node is used by the ImgU driver to output the 3A (auto
focus, auto exposure and auto white balance) statistics for the frames that are
being processed by the ImgU to user space applications. User space applications
can use this statistics data to compute the desired algorithm parameters for
the ImgU.
Configuring the Intel IPU3
==========================
The IPU3 ImgU pipelines can be configured using the Media Controller, defined at
:ref:`media_controller`.
Running mode and firmware binary selection
------------------------------------------
ImgU works based on firmware, currently the ImgU firmware support run 2 pipes
in time-sharing with single input frame data. Each pipe can run at certain mode
- "VIDEO" or "STILL", "VIDEO" mode is commonly used for video frames capture,
and "STILL" is used for still frame capture. However, you can also select
"VIDEO" to capture still frames if you want to capture images with less system
load and power. For "STILL" mode, ImgU will try to use smaller BDS factor and
output larger bayer frame for further YUV processing than "VIDEO" mode to get
high quality images. Besides, "STILL" mode need XNR3 to do noise reduction,
hence "STILL" mode will need more power and memory bandwidth than "VIDEO" mode.
TNR will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is
running at "VIDEO" mode by default, the user can use v4l2 control
V4L2_CID_INTEL_IPU3_MODE (currently defined in
drivers/staging/media/ipu3/include/uapi/intel-ipu3.h) to query and set the
running mode. For user, there is no difference for buffer queueing between the
"VIDEO" and "STILL" mode, mandatory input and main output node should be
enabled and buffers need be queued, the statistics and the view-finder queues
are optional.
The firmware binary will be selected according to current running mode, such log
"using binary if_to_osys_striped " or "using binary if_to_osys_primary_striped"
could be observed if you enable the ImgU dynamic debug, the binary
if_to_osys_striped is selected for "VIDEO" and the binary
"if_to_osys_primary_striped" is selected for "STILL".
Processing the image in raw Bayer format
----------------------------------------
Configuring ImgU V4L2 subdev for image processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ImgU V4L2 subdevs have to be configured with media controller APIs to have
all the video nodes setup correctly.
Let us take "ipu3-imgu 0" subdev as an example.
.. code-block:: none
media-ctl -d $MDEV -r
media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1]
media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1]
media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1]
Also the pipe mode of the corresponding V4L2 subdev should be set as desired
(e.g 0 for video mode or 1 for still mode) through the control id 0x009819a1 as
below.
.. code-block:: none
yavta -w "0x009819A1 1" /dev/v4l-subdev7
Certain hardware blocks in ImgU pipeline can change the frame resolution by
cropping or scaling, these hardware blocks include Input Feeder(IF), Bayer Down
Scaler (BDS) and Geometric Distortion Correction (GDC).
There is also a block which can change the frame resolution - YUV Scaler, it is
only applicable to the secondary output.
RAW Bayer frames go through these ImgU pipeline hardware blocks and the final
processed image output to the DDR memory.
.. kernel-figure:: ipu3_rcb.svg
:alt: ipu3 resolution blocks image
IPU3 resolution change hardware blocks
**Input Feeder**
Input Feeder gets the Bayer frame data from the sensor, it can enable cropping
of lines and columns from the frame and then store pixels into device's internal
pixel buffer which are ready to readout by following blocks.
**Bayer Down Scaler**
Bayer Down Scaler is capable of performing image scaling in Bayer domain, the
downscale factor can be configured from 1X to 1/4X in each axis with
configuration steps of 0.03125 (1/32).
**Geometric Distortion Correction**
Geometric Distortion Correction is used to perform correction of distortions
and image filtering. It needs some extra filter and envelope padding pixels to
work, so the input resolution of GDC should be larger than the output
resolution.
**YUV Scaler**
YUV Scaler which similar with BDS, but it is mainly do image down scaling in
YUV domain, it can support up to 1/12X down scaling, but it can not be applied
to the main output.
The ImgU V4L2 subdev has to be configured with the supported resolutions in all
the above hardware blocks, for a given input resolution.
For a given supported resolution for an input frame, the Input Feeder, Bayer
Down Scaler and GDC blocks should be configured with the supported resolutions
as each hardware block has its own alignment requirement.
You must configure the output resolution of the hardware blocks smartly to meet
the hardware requirement along with keeping the maximum field of view. The
intermediate resolutions can be generated by specific tool -
https://github.com/intel/intel-ipu3-pipecfg
This tool can be used to generate intermediate resolutions. More information can
be obtained by looking at the following IPU3 ImgU configuration table.
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
Under baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
directory, graph_settings_ov5670.xml can be used as an example.
The following steps prepare the ImgU pipeline for the image processing.
1. The ImgU V4L2 subdev data format should be set by using the
VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
2. The ImgU V4L2 subdev cropping should be set by using the
VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target,
using the input feeder height and width.
3. The ImgU V4L2 subdev composing should be set by using the
VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the target,
using the BDS height and width.
For the ov5670 example, for an input frame with a resolution of 2592x1944
(which is input to the ImgU subdev pad 0), the corresponding resolutions
for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
respectively.
Once this is done, the received raw Bayer frames can be input to the ImgU
V4L2 subdev as below, using the open source application v4l2n [#f1]_.
For an image captured with 2592x1944 [#f4]_ resolution, with desired output
resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following
v4l2n command can be used. This helps process the raw Bayer frames and produces
the desired results for the main output image and the viewfinder output, in NV12
format.
.. code-block:: none
v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069 \
--reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1 \
--output=/tmp/frames.out --open=/dev/video5 \
--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \
--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2 \
--output=/tmp/frames.vf --open=/dev/video6 \
--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \
--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7 \
--output=/tmp/frames.3A --fmt=type:META_CAPTURE,? \
--reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5
You can also use yavta [#f2]_ command to do same thing as above:
.. code-block:: none
yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
--file=frame-#.out-f NV12 /dev/video5 & \
yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
--file=frame-#.vf -f NV12 /dev/video6 & \
yavta --data-prefix -Bmeta-capture -c10 -n5 -I \
--file=frame-#.3a /dev/video7 & \
yavta --data-prefix -Boutput-mplane -c10 -n5 -I -s2592x1944 \
--file=/tmp/frame-in.cio2 -f IPU3_SGRBG10 /dev/video4
where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to
input, output, viewfinder and 3A statistics video nodes respectively.
Converting the raw Bayer image into YUV domain
----------------------------------------------
The processed images after the above step, can be converted to YUV domain
as below.
Main output frames
~~~~~~~~~~~~~~~~~~
.. code-block:: none
raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm
where 2560x1920 is output resolution, NV12 is the video format, followed
by input frame and output PNM file.
Viewfinder output frames
~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: none
raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm
where 2560x1920 is output resolution, NV12 is the video format, followed
by input frame and output PNM file.
Example user space code for IPU3
================================
User space code that configures and uses IPU3 is available here.
https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/
The source can be located under hal/intel directory.
Overview of IPU3 pipeline
=========================
IPU3 pipeline has a number of image processing stages, each of which takes a
set of parameters as input. The major stages of pipelines are shown here:
.. kernel-render:: DOT
:alt: IPU3 ImgU Pipeline
:caption: IPU3 ImgU Pipeline Diagram
digraph "IPU3 ImgU" {
node [shape=box]
splines="ortho"
rankdir="LR"
a [label="Raw pixels"]
b [label="Bayer Downscaling"]
c [label="Optical Black Correction"]
d [label="Linearization"]
e [label="Lens Shading Correction"]
f [label="White Balance / Exposure / Focus Apply"]
g [label="Bayer Noise Reduction"]
h [label="ANR"]
i [label="Demosaicing"]
j [label="Color Correction Matrix"]
k [label="Gamma correction"]
l [label="Color Space Conversion"]
m [label="Chroma Down Scaling"]
n [label="Chromatic Noise Reduction"]
o [label="Total Color Correction"]
p [label="XNR3"]
q [label="TNR"]
r [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
s [label="YUV Downscaling"]
t [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
{ rank=same; a -> b -> c -> d -> e -> f -> g -> h -> i }
{ rank=same; j -> k -> l -> m -> n -> o -> p -> q -> s -> t}
a -> j [style=invis, weight=10]
i -> j
q -> r
}
The table below presents a description of the above algorithms.
======================== =======================================================
Name Description
======================== =======================================================
Optical Black Correction Optical Black Correction block subtracts a pre-defined
value from the respective pixel values to obtain better
image quality.
Defined in struct ipu3_uapi_obgrid_param.
Linearization This algo block uses linearization parameters to
address non-linearity sensor effects. The Lookup table
table is defined in
struct ipu3_uapi_isp_lin_vmem_params.
SHD Lens shading correction is used to correct spatial
non-uniformity of the pixel response due to optical
lens shading. This is done by applying a different gain
for each pixel. The gain, black level etc are
configured in struct ipu3_uapi_shd_config_static.
BNR Bayer noise reduction block removes image noise by
applying a bilateral filter.
See struct ipu3_uapi_bnr_static_config for details.
ANR Advanced Noise Reduction is a block based algorithm
that performs noise reduction in the Bayer domain. The
convolution matrix etc can be found in
struct ipu3_uapi_anr_config.
DM Demosaicing converts raw sensor data in Bayer format
into RGB (Red, Green, Blue) presentation. Then add
outputs of estimation of Y channel for following stream
processing by Firmware. The struct is defined as
struct ipu3_uapi_dm_config.
Color Correction Color Correction algo transforms sensor specific color
space to the standard "sRGB" color space. This is done
by applying 3x3 matrix defined in
struct ipu3_uapi_ccm_mat_config.
Gamma correction Gamma correction struct ipu3_uapi_gamma_config is a
basic non-linear tone mapping correction that is
applied per pixel for each pixel component.
CSC Color space conversion transforms each pixel from the
RGB primary presentation to YUV (Y: brightness,
UV: Luminance) presentation. This is done by applying
a 3x3 matrix defined in
struct ipu3_uapi_csc_mat_config
CDS Chroma down sampling
After the CSC is performed, the Chroma Down Sampling
is applied for a UV plane down sampling by a factor
of 2 in each direction for YUV 4:2:0 using a 4x2
configurable filter struct ipu3_uapi_cds_params.
CHNR Chroma noise reduction
This block processes only the chrominance pixels and
performs noise reduction by cleaning the high
frequency noise.
See struct struct ipu3_uapi_yuvp1_chnr_config.
TCC Total color correction as defined in struct
struct ipu3_uapi_yuvp2_tcc_static_config.
XNR3 eXtreme Noise Reduction V3 is the third revision of
noise reduction algorithm used to improve image
quality. This removes the low frequency noise in the
captured image. Two related structs are being defined,
struct ipu3_uapi_isp_xnr3_params for ISP data memory
and struct ipu3_uapi_isp_xnr3_vmem_params for vector
memory.
TNR Temporal Noise Reduction block compares successive
frames in time to remove anomalies / noise in pixel
values. struct ipu3_uapi_isp_tnr3_vmem_params and
struct ipu3_uapi_isp_tnr3_params are defined for ISP
vector and data memory respectively.
======================== =======================================================
Other often encountered acronyms not listed in above table:
ACC
Accelerator cluster
AWB_FR
Auto white balance filter response statistics
BDS
Bayer downscaler parameters
CCM
Color correction matrix coefficients
IEFd
Image enhancement filter directed
Obgrid
Optical black level compensation
OSYS
Output system configuration
ROI
Region of interest
YDS
Y down sampling
YTM
Y-tone mapping
A few stages of the pipeline will be executed by firmware running on the ISP
processor, while many others will use a set of fixed hardware blocks also
called accelerator cluster (ACC) to crunch pixel data and produce statistics.
ACC parameters of individual algorithms, as defined by
struct ipu3_uapi_acc_param, can be chosen to be applied by the user
space through struct struct ipu3_uapi_flags embedded in
struct ipu3_uapi_params structure. For parameters that are configured as
not enabled by the user space, the corresponding structs are ignored by the
driver, in which case the existing configuration of the algorithm will be
preserved.
References
==========
.. [#f1] https://github.com/intel/nvt
.. [#f2] http://git.ideasonboard.org/yavta.git
.. [#f4] ImgU limitation requires an additional 16x16 for all input resolutions
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
Intel IPU3 ImgU driver
1-10이 문서는 GPL-2.0 라이선스를 따르는 Intel Image Processing Unit 3(IPU3) Imaging Unit(ImgU) driver 안내입니다. Copyright (C) 2018 Intel Corporation.
CIO2와 ImgU 소개
11-39이 문서는 `drivers/media/pci/intel/ipu3`의 Intel IPU3 CIO2 driver와 `drivers/staging/media/ipu3`의 ImgU driver를 설명합니다.
일부 Kaby Lake 및 Sky Lake U/Y processor platform의 Intel IPU3는 Imaging Unit(ImgU)과 MIPI CSI-2 receiver인 CIO2 device, 두 부분으로 구성됩니다.
CIO2 device는 sensor에서 raw Bayer data를 받아 IPU3 ImgU가 소비하는 IPU3 전용 format으로 frame을 출력합니다. Driver source는 `drivers/media/pci/intel/ipu3/ipu3-cio2*`이며 `CONFIG_VIDEO_IPU3_CIO2` config option으로 활성화합니다.
ImgU는 IPU3 CIO2가 capture한 image를 처리합니다. Driver source는 `drivers/staging/media/ipu3`에 있고 `CONFIG_VIDEO_IPU3_IMGU` config option으로 활성화합니다.
두 driver module의 이름은 각각 `ipu3_csi2`와 `ipu3_imgu`입니다. Driver는 Kaby Lake U/Y processor platform에서 시험되었습니다.
두 driver 모두 V4L2, Media Controller, V4L2 sub-device interface를 구현합니다. IPU3 CIO2 driver는 V4L2 sub-device sensor driver를 통해 CIO2 MIPI CSI-2 interface에 연결된 camera sensor를 지원합니다.
CIO2가 sensor frame을 IPU3 전용 packed raw Bayer로 capture하고 ImgU가 이를 YUV와 3A output으로 처리합니다.
CIO2 channel과 pad
40-70CIO2는 user space에 V4L2 subdev interface를 제공하는 단일 V4L2 subdev로 표현됩니다. CSI-2 receiver마다 video node가 하나씩 있고 device 전체에는 Media Controller interface 하나가 있습니다.
CIO2에는 서로 독립적인 capture channel 네 개가 있으며 각 channel은 자체 MIPI CSI-2 receiver와 DMA engine을 가집니다. 각 channel은 V4L2 sub-device node로 노출되는 V4L2 sub-device이며 pad 두 개를 가집니다.
| Pad | 방향 | 용도 |
|---|---|---|
| 0 | sink | Sensor subdev와 연결되는 MIPI CSI-2 input |
| 1 | source | V4L2 video interface와 연결되는 raw video capture |
V4L2 video interface는 DMA engine을 model하며 user space에는 V4L2 video device node로 노출됩니다.
각 channel은 sensor sink pad와 DMA-backed raw capture source pad를 독립적으로 가집니다.
Raw Bayer frame capture
71-133CIO2 MIPI CSI-2 receiver는 CSI-2 port에 연결된 raw sensor에서 IPU3 전용 packed raw Bayer format으로 frame을 capture합니다. Capture frame은 ImgU driver의 input으로 사용됩니다.
IPU3에는 고유한 요구 사항이 있습니다. CSI-2 receiver가 IPU3 전용 packed raw Bayer를 출력하고 여러 video node를 동시에 운용해야 하므로 image processing에는 `raw2pnm` `f1`과 `yavta` `f2` 같은 도구가 필요합니다.
다음은 CSI-2 port 0에 연결된 OV5670 sensor에서 2592x1944 image를 capture하는 예입니다. Media Controller API로 OV5670이 packed raw Bayer frame을 IPU3 CSI-2 receiver에 보내도록 설정합니다.
# This example assumes /dev/media0 as the CIO2 media device
export MDEV=/dev/media0
# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036")
# Establish the link for the media devices using media-ctl
media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]"
# Set the format for the media devices
media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]"
media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]"
media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]"Media link와 세 pad의 `SGRBG10/2592x1944` format을 맞춘 뒤 CIO2 DMA node에서 IPU3 전용 frame을 capture합니다.
Media pipeline을 구성한 뒤 exposure와 gain 같은 sensor별 설정을 `yavta`로 지정할 수 있습니다.
yavta -w 0x009e0903 444 $SDEV
yavta -w 0x009e0913 1024 $SDEV
yavta -w 0x009e0911 2046 $SDEV원하는 sensor 설정을 적용한 뒤 다음과 같이 frame을 capture합니다.
yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \
-f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0")이 명령은 2592x1944 resolution의 sGRBG10 frame 10개를 capture해 `IPU3_SGRBG10` format의 `/tmp/frame-#.bin` file로 출력합니다.
ImgU pipe와 pad
134-180ImgU는 두 V4L2 subdev로 표현되며 각각 user space에 V4L2 subdev interface를 제공합니다.
각 V4L2 subdev는 최대 stream 두 개를 지원하는 pipe 하나를 나타냅니다. 이를 통해 Continuous View Finder(CVF)와 Snapshot During Video(SDV) 같은 고급 camera 기능을 지원합니다.
ImgU에는 서로 독립적인 pipe 두 개가 있고 각각 V4L2 sub-device node로 노출됩니다. 각 pipe는 sink pad 두 개와 source pad 세 개를 가집니다.
| Pad | 방향 | 용도 |
|---|---|---|
| 0 | sink | Input raw video stream |
| 1 | sink | Processing parameter |
| 2 | source | 처리된 main video stream output |
| 3 | source | Viewfinder video stream output |
| 4 | source | 3A statistics |
각 pad는 대응하는 V4L2 video interface와 연결되며 user space에는 V4L2 video device node로 노출됩니다.
Raw frame과 parameter를 받는 두 sink에서 main, viewfinder, 3A의 세 source로 처리 결과가 나옵니다.
ImgU device operation
181-195Input video node(`ipu3-imgu 0/1`:0, `<entity>:<pad-number>` 형식)에 IPU3 전용 packed raw Bayer buffer를 queue하면 ImgU가 처리를 시작해 각 output node에 YUV video와 statistics를 생성합니다.
Input video node에 buffer를 queue할 때 parameter, output, statistics node에도 사용할 buffer가 준비되어 있어야 합니다.
IPU3가 image processing을 시작하려면 최소한 input, main output, 3A statistics, viewfinder video node를 모두 활성화해야 합니다. 각 ImgU V4L2 subdev는 아래의 video node set을 가집니다.
Input·output·viewfinder node
196-211Input video node가 받은 IPU3 전용 packed raw Bayer frame은 IPU3 Imaging Unit에서 처리되고, 목적이 다른 main output과 viewfinder output의 두 video node로 출력됩니다.
IPU3 전용 Bayer format의 상세 내용은 `v4l2-pix-fmt-ipu3-sbggr10`을 참조하십시오.
Driver는 `devices`에 정의된 V4L2 Video Capture Interface를 지원하며 multi-planar API만 지원합니다. 자세한 내용은 `planar-apis`를 참조하십시오.
Parameter video node
212-220Parameter video node는 ImgU algorithm이 image를 처리하는 방식을 설정하는 ImgU algorithm parameter를 받습니다.
IPU3 전용 processing parameter의 상세 내용은 `v4l2-meta-fmt-params`를 참조하십시오.
3A statistics video node
221-229ImgU driver는 처리 중인 frame의 3A, 즉 auto focus·auto exposure·auto white balance statistics를 3A statistics video node를 통해 user space application에 출력합니다.
User space application은 이 statistics data로 ImgU에 적용할 algorithm parameter를 계산할 수 있습니다.
Intel IPU3 구성
230-235IPU3 ImgU pipeline은 `media_controller`에 정의된 Media Controller를 사용해 구성할 수 있습니다.
Running mode와 firmware binary
236-263ImgU는 firmware를 기반으로 동작합니다. 현재 firmware는 하나의 input frame data를 time-sharing하는 pipe 두 개를 지원합니다. 각 pipe는 `VIDEO` 또는 `STILL` mode로 동작할 수 있습니다.
`VIDEO`는 일반적인 video frame capture에 사용하고 `STILL`은 still frame capture에 사용합니다. System load와 power를 줄이고 싶다면 still frame에도 `VIDEO`를 선택할 수 있습니다.
`STILL` mode는 더 높은 image 품질을 위해 더 작은 BDS factor와 더 큰 Bayer frame을 YUV processing에 사용하며 XNR3 noise reduction도 필요합니다. 따라서 `VIDEO`보다 power와 memory bandwidth를 더 사용합니다.
TNR은 `VIDEO` mode에서 활성화되고 `STILL` mode에서는 bypass됩니다. 기본값은 `VIDEO`입니다.
User는 `drivers/staging/media/ipu3/include/uapi/intel-ipu3.h`에 정의된 `V4L2_CID_INTEL_IPU3_MODE` control로 mode를 조회하고 설정할 수 있습니다. `VIDEO`와 `STILL`의 buffer queue 방식은 같으며 input과 main output node는 필수이고 statistics와 viewfinder queue는 선택 사항입니다.
Firmware binary는 현재 mode에 따라 선택됩니다. Dynamic debug를 켜면 `using binary if_to_osys_striped` 또는 `using binary if_to_osys_primary_striped` log를 볼 수 있습니다. `VIDEO`는 `if_to_osys_striped`, `STILL`은 `if_to_osys_primary_striped`를 사용합니다.
Mode에 따라 noise reduction과 firmware binary, resource 사용량이 달라집니다.
Raw Bayer image processing
264-266다음 절은 raw Bayer image를 ImgU pipeline에 입력해 처리하는 구성을 설명합니다.
ImgU subdev와 link 설정
267-290모든 video node를 올바르게 연결하려면 Media Controller API로 ImgU V4L2 subdev를 구성해야 합니다. 다음은 `ipu3-imgu 0` subdev 예입니다.
media-ctl -d $MDEV -r
media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1]
media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1]
media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1]대응하는 V4L2 subdev의 pipe mode도 control ID `0x009819a1`을 통해 원하는 값으로 설정해야 합니다. `0`은 video mode, `1`은 still mode입니다.
yavta -w "0x009819A1 1" /dev/v4l-subdev7Resolution 변경 hardware block
291-349ImgU pipeline에서 Input Feeder(IF), Bayer Down Scaler(BDS), Geometric Distortion Correction(GDC)은 crop 또는 scaling으로 frame resolution을 바꿀 수 있습니다. YUV Scaler도 resolution을 바꾸지만 secondary output에만 적용됩니다.
Raw Bayer frame은 이 hardware block들을 거쳐 처리된 뒤 최종 image가 DDR memory로 출력됩니다.
원문 figure source는 `Documentation/admin-guide/media/ipu3_rcb.svg`입니다.
Main output은 GDC 뒤에서 나오며 secondary output만 YUV Scaler를 추가로 사용할 수 있습니다.
Input Feeder는 sensor의 Bayer frame을 받아 row와 column을 crop하고 뒤 block이 읽을 수 있도록 device 내부 pixel buffer에 pixel을 저장합니다.
Bayer Down Scaler는 Bayer domain에서 image scaling을 수행합니다. 각 axis의 downscale factor는 1X부터 1/4X까지 0.03125(1/32) step으로 설정할 수 있습니다.
GDC는 distortion correction과 image filtering을 수행합니다. Filter와 envelope padding pixel이 추가로 필요하므로 GDC input resolution은 output resolution보다 커야 합니다.
YUV Scaler는 BDS와 비슷하지만 YUV domain에서 주로 downscaling을 수행하며 최대 1/12X까지 지원합니다. Main output에는 적용할 수 없습니다.
주어진 input resolution에 대해 ImgU V4L2 subdev의 모든 hardware block을 지원 resolution으로 구성해야 합니다. IF, BDS, GDC는 각자 alignment requirement가 있으므로 각 block의 output resolution을 hardware requirement와 최대 field of view를 함께 만족하도록 선택해야 합니다.
- Intel IPU3 pipeline configuration tool
https://github.com/intel/intel-ipu3-pipecfg - ChromiumOS board overlay examples
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
`intel-ipu3-pipecfg`로 intermediate resolution을 생성할 수 있습니다. IPU3 ImgU 구성 예는 ChromiumOS overlay의 `baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss/graph_settings_ov5670.xml`을 참조하십시오.
ImgU format·crop·compose 설정
350-376ImgU pipeline을 image processing에 준비시키는 순서는 다음과 같습니다.
- Pad 0에서 `VIDIOC_SUBDEV_S_FMT`를 호출해 위에서 구한 GDC width와 height로 ImgU V4L2 subdev data format을 설정합니다.
- Pad 0에서 target을 `V4L2_SEL_TGT_CROP`으로 지정한 `VIDIOC_SUBDEV_S_SELECTION`을 호출해 Input Feeder height와 width로 crop을 설정합니다.
- Pad 0에서 target을 `V4L2_SEL_TGT_COMPOSE`로 지정한 `VIDIOC_SUBDEV_S_SELECTION`을 호출해 BDS height와 width로 compose를 설정합니다.
Format은 GDC resolution, crop은 IF resolution, compose는 BDS resolution을 사용합니다.
OV5670 예에서 ImgU subdev pad 0으로 들어오는 2592x1944 input frame에 대응하는 IF, BDS, GDC resolution은 각각 2592x1944, 2592x1944, 2560x1920입니다.
이 설정을 마치면 open source application `v4l2n` `f1`을 사용해 수신한 raw Bayer frame을 ImgU V4L2 subdev에 입력할 수 있습니다.
2592x1944 `f4` input을 main output 2560x1920, viewfinder 2560x1920, NV12 format으로 처리하는 명령은 다음 절에 나옵니다.
동시 queue와 frame 처리
377-406v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069 \
--reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1 \
--output=/tmp/frames.out --open=/dev/video5 \
--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \
--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2 \
--output=/tmp/frames.vf --open=/dev/video6 \
--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \
--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7 \
--output=/tmp/frames.3A --fmt=type:META_CAPTURE,? \
--reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5같은 작업은 `yavta` `f2`로도 수행할 수 있습니다.
yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
--file=frame-#.out-f NV12 /dev/video5 & \
yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
--file=frame-#.vf -f NV12 /dev/video6 & \
yavta --data-prefix -Bmeta-capture -c10 -n5 -I \
--file=frame-#.3a /dev/video7 & \
yavta --data-prefix -Boutput-mplane -c10 -n5 -I -s2592x1944 \
--file=/tmp/frame-in.cio2 -f IPU3_SGRBG10 /dev/video4`/dev/video4`, `/dev/video5`, `/dev/video6`, `/dev/video7`은 각각 input, main output, viewfinder, 3A statistics video node입니다.
Input buffer를 queue하기 전에 main, viewfinder, 3A output queue를 준비하고 네 pipe를 함께 stream합니다.
Raw Bayer에서 YUV domain으로
407-412앞 단계에서 처리한 image는 다음과 같이 YUV domain의 output file에서 PNM으로 변환할 수 있습니다.
Main output frame 변환
413-422raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm`2560x1920`은 output resolution, `NV12`는 video format이며 그 뒤에 input frame과 output PNM file이 옵니다.
Viewfinder output frame 변환
423-432raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm`2560x1920`은 output resolution, `NV12`는 video format이며 그 뒤에 input frame과 output PNM file이 옵니다.
IPU3 user space code
433-441IPU3를 구성하고 사용하는 user space code는 다음 ChromiumOS source에 있습니다.
관련 source는 `hal/intel` directory에 있습니다.
IPU3 ISP pipeline
442-485IPU3 pipeline에는 여러 image processing stage가 있으며 각 stage는 parameter set을 input으로 받습니다. 주요 stage는 원문 DOT diagram과 같습니다.
Raw/Bayer stage가 YUV stage로 이어지고 TNR 뒤에서 main DDR과 YUV downscaled DDR output으로 분기합니다.
ISP algorithm 설명
486-553각 algorithm의 역할과 관련 UAPI structure는 다음과 같습니다.
| 이름 | 설명 |
|---|---|
| Optical Black Correction | 미리 정의한 값을 해당 pixel에서 빼 image 품질을 개선합니다. `struct ipu3_uapi_obgrid_param`에 정의됩니다. |
| Linearization | Linearization parameter로 sensor의 non-linearity effect를 보정합니다. Lookup table은 `struct ipu3_uapi_isp_lin_vmem_params`에 정의됩니다. |
| SHD | Lens shading에 따른 pixel response의 공간적 불균일성을 pixel별 gain으로 보정합니다. Gain과 black level 등은 `struct ipu3_uapi_shd_config_static`에 설정합니다. |
| BNR | Bilateral filter를 적용해 Bayer image noise를 제거합니다. `struct ipu3_uapi_bnr_static_config`를 참조하십시오. |
| ANR | Bayer domain에서 noise reduction을 수행하는 block-based algorithm입니다. Convolution matrix 등은 `struct ipu3_uapi_anr_config`에 있습니다. |
| DM | Demosaicing으로 Bayer raw sensor data를 RGB로 변환하고 뒤 firmware stream processing을 위한 Y channel 추정 output을 추가합니다. `struct ipu3_uapi_dm_config`에 정의됩니다. |
| Color Correction | Sensor 전용 color space를 표준 sRGB로 변환합니다. `struct ipu3_uapi_ccm_mat_config`의 3x3 matrix를 적용합니다. |
| Gamma correction | `struct ipu3_uapi_gamma_config`에 정의된 기본 non-linear tone mapping correction을 pixel component별로 적용합니다. |
| CSC | RGB primary 표현을 YUV(Y: brightness, UV: luminance) 표현으로 변환합니다. `struct ipu3_uapi_csc_mat_config`의 3x3 matrix를 적용합니다. |
| CDS | CSC 뒤 YUV 4:2:0 UV plane을 각 방향으로 2배 downsample합니다. `struct ipu3_uapi_cds_params`의 configurable 4x2 filter를 사용합니다. |
| CHNR | Chrominance pixel만 처리해 high-frequency noise를 줄입니다. `struct ipu3_uapi_yuvp1_chnr_config`를 참조하십시오. |
| TCC | Total color correction은 `struct ipu3_uapi_yuvp2_tcc_static_config`에 정의됩니다. |
| XNR3 | eXtreme Noise Reduction V3는 capture image의 low-frequency noise를 제거합니다. ISP data memory용 `struct ipu3_uapi_isp_xnr3_params`와 vector memory용 `struct ipu3_uapi_isp_xnr3_vmem_params`가 정의됩니다. |
| TNR | 연속 frame을 시간 축에서 비교해 pixel anomaly와 noise를 제거합니다. ISP vector memory용 `struct ipu3_uapi_isp_tnr3_vmem_params`와 data memory용 `struct ipu3_uapi_isp_tnr3_params`가 정의됩니다. |
추가 약어
554-576위 표에 없지만 자주 등장하는 약어는 다음과 같습니다.
| 약어 | 의미 |
|---|---|
| ACC | Accelerator cluster |
| AWB_FR | Auto white balance filter response statistics |
| BDS | Bayer downscaler parameter |
| CCM | Color correction matrix coefficient |
| IEFd | Image enhancement filter directed |
| Obgrid | Optical black level compensation |
| OSYS | Output system configuration |
| ROI | Region of interest |
| YDS | Y down sampling |
| YTM | Y-tone mapping |
Firmware와 ACC parameter
577-588Pipeline의 일부 stage는 ISP processor에서 동작하는 firmware가 실행하고, 많은 stage는 accelerator cluster(ACC)라고도 하는 fixed hardware block set이 pixel data를 처리하고 statistics를 생성합니다.
`struct ipu3_uapi_acc_param`에 정의된 개별 algorithm의 ACC parameter는 `struct ipu3_uapi_params`에 포함된 `struct ipu3_uapi_flags`를 통해 user space가 적용 여부를 선택할 수 있습니다.
User space가 enable하지 않은 parameter에 대응하는 structure는 driver가 무시하며, 그 경우 기존 algorithm configuration이 유지됩니다.
Reference와 제한 사항
589-596ImgU limitation 때문에 모든 input resolution에는 추가 16x16 pixel이 필요합니다(`f4`).
IPU3 capture-to-ISP architecture
ipu3.rst:1-596Camera sensor에서 CIO2 packed raw Bayer capture, ImgU의 input·main·viewfinder·3A 동시 queue, IF/BDS/GDC resolution 설정, raw-to-YUV ISP stage까지 이어지는 전체 구현 흐름입니다.
CIO2와 ImgU는 별도 driver지만 DDR의 IPU3 전용 raw frame을 사이에 두고 하나의 capture-to-processing workflow를 이룹니다.