Network Filesystem Services Library

netfslib의 범위와 세 객체

Network filesystem services library인 netfslib는 network filesystem이 VM/VFS API operation을 구현하도록 돕는 함수 집합입니다. 일반 buffered read, readahead, write, writeback을 맡고 unbuffered 및 direct I/O도 처리합니다.

I/O 크기 재협상과 실패한 I/O retry, local caching을 지원하며 미래에는 content encryption도 제공할 예정입니다. VM interface 변경에서 파일시스템을 가능한 한 격리하고 large multipage folio 같은 VM 기능을 처리하므로 파일시스템은 기본적으로 read·write RPC 수단만 제공하면 됩니다.

내부 I/O는 request, stream, subrequest 세 객체로 구성됩니다. Request는 전체 I/O 진행과 resource를 추적하며 결과 집계도 request level에서 수행합니다. 하나의 request I/O는 여러 parallel stream의 subrequest로 나뉩니다.

Stream은 서로 겹치지 않는 subrequest의 연속이며 내부 subrequest가 반드시 contiguous일 필요는 없습니다. Subrequest는 기본 I/O 단위로 RPC 하나 또는 cache I/O 하나를 나타내며 library가 filesystem이나 cache에 실행을 맡깁니다.

.. SPDX-License-Identifier: GPL-2.0

===================================
Network Filesystem Services Library
===================================

.. Contents:

 - Overview.
   - Requests and streams.
   - Subrequests.
   - Result collection and retry.
   - Local caching.
   - Content encryption (fscrypt).
 - Per-inode context.
   - Inode context helper functions.
   - Inode locking.
   - Inode writeback.
 - High-level VFS API.
   - Unlocked read/write iter.
   - Pre-locked read/write iter.
   - Monolithic files API.
   - Memory-mapped I/O API.
 - High-level VM API.
   - Deprecated PG_private2 API.
 - I/O request API.
   - Request structure.
   - Stream structure.
   - Subrequest structure.
   - Filesystem methods.
   - Terminating a subrequest.
   - Local cache API.
 - API function reference.


Overview
========

The network filesystem services library, netfslib, is a set of functions
designed to aid a network filesystem in implementing VM/VFS API operations.  It
takes over the normal buffered read, readahead, write and writeback and also
handles unbuffered and direct I/O.

The library provides support for (re-)negotiation of I/O sizes and retrying
failed I/O as well as local caching and will, in the future, provide content
encryption.

It insulates the filesystem from VM interface changes as much as possible and
handles VM features such as large multipage folios.  The filesystem basically
just has to provide a way to perform read and write RPC calls.

The way I/O is organised inside netfslib consists of a number of objects:

 * A *request*.  A request is used to track the progress of the I/O overall and
   to hold on to resources.  The collection of results is done at the request
   level.  The I/O within a request is divided into a number of parallel
   streams of subrequests.

 * A *stream*.  A non-overlapping series of subrequests.  The subrequests
   within a stream do not have to be contiguous.

 * A *subrequest*.  This is the basic unit of I/O.  It represents a single RPC
   call or a single cache I/O operation.  The library passes these to the
   filesystem and the cache to perform.

Requests and Streams

읽기와 쓰기 stream 구성

Page cache에 복사만 하는 경우가 아니라 실제 I/O를 수행할 때 netfslib는 진행 추적과 resource 보유를 위해 request를 하나 이상 만듭니다.

Read operation은 stream 하나만 사용하며 그 안에는 RPC와 cache subrequest처럼 source가 다른 subrequest가 섞일 수 있습니다.

Write operation은 destination마다 별도 stream을 둘 수 있습니다. 예를 들어 local cache용 stream과 server용 stream이 있습니다. 현재 최대 두 stream만 허용하지만 여러 server에 병렬 write가 필요하면 늘릴 수 있습니다.

한 write stream의 subrequest alignment·size는 다른 stream과 맞을 필요가 없습니다. Netfslib가 각 stream에서 source buffer 위 subrequest tiling을 독립 수행하고, 한 stream에는 다른 stream과 대응하지 않는 hole도 있을 수 있습니다.

Subrequest는 source·destination buffer의 folio 또는 vector boundary와도 맞을 필요가 없습니다. Library가 결과 집계와 folio flag·reference 처리를 담당합니다.

--------------------

When actually performing I/O (as opposed to just copying into the pagecache),
netfslib will create one or more requests to track the progress of the I/O and
to hold resources.

A read operation will have a single stream and the subrequests within that
stream may be of mixed origins, for instance mixing RPC subrequests and cache
subrequests.

On the other hand, a write operation may have multiple streams, where each
stream targets a different destination.  For instance, there may be one stream
writing to the local cache and one to the server.  Currently, only two streams
are allowed, but this could be increased if parallel writes to multiple servers
is desired.

The subrequests within a write stream do not need to match alignment or size
with the subrequests in another write stream and netfslib performs the tiling
of subrequests in each stream over the source buffer independently.  Further,
each stream may contain holes that don't correspond to holes in the other
stream.

In addition, the subrequests do not need to correspond to the boundaries of the
folios or vectors in the source/destination buffer.  The library handles the
collection of results and the wrangling of folio flags and references.

Subrequests

Subrequest 협상과 발행

Subrequest는 netfslib와 사용하는 filesystem 사이 상호작용의 중심입니다. 각 subrequest는 read·write RPC 하나 또는 cache operation 하나에 대응하며 library가 여러 결과를 이어 붙여 고수준 operation을 만듭니다.

설정 과정은 두 단계입니다. 선택적 prepare 단계에서 filesystem은 최대 byte 수와 최대 vector 수를 제한할 수 있습니다. RDMA 제한이나 CIFS credit 획득처럼 server와의 협상이 필요할 수 있습니다. 다음 issue 단계에서 subrequest를 filesystem에 넘겨 실제 실행합니다.

Read는 VM/VFS가 요청 크기를 미리 알려주므로 library가 최대값을 설정한 뒤 cache, filesystem 순으로 줄일 수 있습니다. Cache가 read를 수행할 의사가 있는지도 filesystem보다 먼저 확인합니다.

Writeback은 page cache를 걸어보기 전까지 쓸 양을 모르므로 library가 한도를 설정하지 않습니다.

완료 시 filesystem 또는 cache가 library에 알리고 collection이 시작됩니다. Sync request는 application thread, async request는 work queue에서 결과를 집계합니다.

-----------

Subrequests are at the heart of the interaction between netfslib and the
filesystem using it.  Each subrequest is expected to correspond to a single
read or write RPC or cache operation.  The library will stitch together the
results from a set of subrequests to provide a higher level operation.

Netfslib has two interactions with the filesystem or the cache when setting up
a subrequest.  First, there's an optional preparatory step that allows the
filesystem to negotiate the limits on the subrequest, both in terms of maximum
number of bytes and maximum number of vectors (e.g. for RDMA).  This may
involve negotiating with the server (e.g. cifs needing to acquire credits).

And, secondly, there's the issuing step in which the subrequest is handed off
to the filesystem to perform.

Note that these two steps are done slightly differently between read and write:

 * For reads, the VM/VFS tells us how much is being requested up front, so the
   library can preset maximum values that the cache and then the filesystem can
   then reduce.  The cache also gets consulted first on whether it wants to do
   a read before the filesystem is consulted.

 * For writeback, it is unknown how much there will be to write until the
   pagecache is walked, so no limit is set by the library.

Once a subrequest is completed, the filesystem or cache informs the library of
the completion and then collection is invoked.  Depending on whether the
request is synchronous or asynchronous, the collection of results will be done
in either the application thread or in a work queue.

Result Collection and Retry

결과 집계와 실패 구간 retiling

Subrequest가 끝날 때마다 library가 결과를 모아 정리하고 적절하면 folio를 점진적으로 unlock합니다. Request가 끝나면 필요 시 async completion을 호출합니다. Filesystem이 중간 진행을 알려 더 일찍 folio를 unlock하게 할 수도 있습니다.

Subrequest가 실패하면 netfslib가 retry할 수 있습니다. 모든 subrequest 완료를 기다린 뒤 filesystem이 request resource·state와 subrequest를 조정할 기회를 주고 다시 prepare·issue합니다.

이 과정에서 stream 안의 연속 실패 subrequest tiling을 바꾸고 필요에 따라 subrequest를 추가하거나 남는 것을 버릴 수 있습니다. Network size가 달라지거나 server가 더 작은 chunk를 요구하는 경우가 예입니다.

연속된 cache-read subrequest 하나 이상이 실패하면 library가 filesystem 실행으로 전환합니다. Cache parameter가 아니라 filesystem parameter에 맞게 다시 협상하고 retile합니다.

---------------------------

As subrequests complete, the results are collected and collated by the library
and folio unlocking is performed progressively (if appropriate).  Once the
request is complete, async completion will be invoked (again, if appropriate).
It is possible for the filesystem to provide interim progress reports to the
library to cause folio unlocking to happen earlier if possible.

If any subrequests fail, netfslib can retry them.  It will wait until all
subrequests are completed, offer the filesystem the opportunity to fiddle with
the resources/state held by the request and poke at the subrequests before
re-preparing and re-issuing the subrequests.

This allows the tiling of contiguous sets of failed subrequest within a stream
to be changed, adding more subrequests or ditching excess as necessary (for
instance, if the network sizes change or the server decides it wants smaller
chunks).

Further, if one or more contiguous cache-read subrequests fail, the library
will pass them to the filesystem to perform instead, renegotiating and retiling
them as necessary to fit with the filesystem's parameters rather than those of
the cache.

Local Caching

fscache와 cache-only writeback

Netfslib는 `fscache`를 통해 network filesystem에서 얻거나 쓴 데이터 사본을 local disk에 cache할 수 있습니다. `netfs_inode`에 cookie가 연결되면 library가 저장·조회와 일부 invalidation을 자동 관리합니다.

과거에는 cache에 write 중인 page를 추적하려고 `PG_private_2`, 별칭 `PG_fscache`를 사용했지만 `PG_private_2` 제거 예정으로 이 방식은 deprecated되었습니다.

대신 cache에 데이터가 없어 server에서 읽은 folio를 dirty로 표시하고 `folio->private`에 `NETFS_FOLIO_COPY_TO_CACHE`를 설정하여 writeback이 cache에 쓰게 합니다. 그 전에 folio가 수정되면 special value를 clear하고 일반 dirty write가 됩니다.

Writeback에서 이 표시가 있는 folio는 server가 아니라 cache에만 씁니다. Cache-only와 server-and-cache write가 섞이면 두 stream을 사용합니다. Cache stream과 server stream을 따로 보내며 server stream에는 cache-only folio 위치에 hole이 생깁니다.

-------------

One of the services netfslib provides, via ``fscache``, is the option to cache
on local disk a copy of the data obtained from/written to a network filesystem.
The library will manage the storing, retrieval and some invalidation of data
automatically on behalf of the filesystem if a cookie is attached to the
``netfs_inode``.

Note that local caching used to use the PG_private_2 (aliased as PG_fscache) to
keep track of a page that was being written to the cache, but this is now
deprecated as PG_private_2 will be removed.

Instead, folios that are read from the server for which there was no data in
the cache will be marked as dirty and will have ``folio->private`` set to a
special value (``NETFS_FOLIO_COPY_TO_CACHE``) and left to writeback to write.
If the folio is modified before that happened, the special value will be
cleared and the write will become normally dirty.

When writeback occurs, folios that are so marked will only be written to the
cache and not to the server.  Writeback handles mixed cache-only writes and
server-and-cache writes by using two streams, sending one to the cache and one
to the server.  The server stream will have gaps in it corresponding to those
folios.

Content Encryption (fscrypt)

향후 client-side content encryption

현재는 아직 구현하지 않지만 netfslib는 미래에 network filesystem을 대신해 client-side content encryption을 수행할 예정입니다. Ceph 같은 경우 적절하면 `fscrypt`를 사용할 수 있지만 CIFS처럼 적합하지 않은 경우도 있습니다.

Local cache에는 server에 쓰는 데이터와 같은 암호화 방식으로 encrypted data를 저장합니다. Library는 필요에 따라 bounce buffering과 read-modify-write cycle을 적용합니다.

----------------------------

Though it does not do so yet, at some point netfslib will acquire the ability
to do client-side content encryption on behalf of the network filesystem (Ceph,
for example).  fscrypt can be used for this if appropriate (it may not be -
cifs, for example).

The data will be stored encrypted in the local cache using the same manner of
encryption as the data written to the server and the library will impose bounce
buffering and RMW cycles as necessary.


Per-Inode Context

netfs_inode embedding과 상태 flag

Netfslib는 관리하는 각 netfs inode에 상태를 저장할 공간이 필요해 `struct netfs_inode`를 정의합니다. Network filesystem은 inode wrapper에서 VFS `struct inode` 대신 이 구조체를 포함해야 합니다. 예시 `struct my_inode`는 `struct netfs_inode netfs`를 embedded합니다.

그러면 netfslib가 inode pointer에서 `container_of()`로 상태를 찾을 수 있어 helper를 VFS/VM operation table에 직접 연결할 수 있습니다.

`inode`는 VFS inode, `ops`는 filesystem이 netfslib에 제공하는 operation set입니다. `cache`는 local cache cookie이며 caching이 없거나 fscache가 disabled이면 NULL 또는 field 자체가 없습니다.

`remote_i_size`는 server의 file size입니다. Local modification이 아직 writeback되지 않았으면 `inode->i_size`와 다를 수 있습니다.

`flags` 중 `NETFS_ICTX_MODIFIED_ATTR`은 netfslib가 mtime/ctime을 수정했음을 나타내며 filesystem이 무시하거나 clear할 수 있습니다.

`NETFS_ICTX_UNBUFFERED`는 alignment 제한 없는 direct I/O 유사 unbuffered I/O를 사용합니다. 필요하면 RMW를 하고 mmap도 사용하지 않는 한 page cache를 쓰지 않습니다. `NETFS_ICTX_WRITETHROUGH`는 buffered write가 page cache에 들어갈 때 I/O를 설정·dispatch하며 mmap은 일반 writeback을 사용합니다.

`NETFS_ICTX_SINGLE_NO_UPLOAD`는 AFS directory처럼 content 전체를 한 번에 읽어야 하고 server에는 writeback하면 안 되는 monolithic file을 표시합니다. Local cache에는 저장할 수 있습니다.

=================

The network filesystem helper library needs a place to store a bit of state for
its use on each netfs inode it is helping to manage.  To this end, a context
structure is defined::

        struct netfs_inode {
                struct inode inode;
                const struct netfs_request_ops *ops;
                struct fscache_cookie * cache;
                loff_t remote_i_size;
                unsigned long flags;
                ...
        };

A network filesystem that wants to use netfslib must place one of these in its
inode wrapper struct instead of the VFS ``struct inode``.  This can be done in
a way similar to the following::

        struct my_inode {
                struct netfs_inode netfs; /* Netfslib context and vfs inode */
                ...
        };

This allows netfslib to find its state by using ``container_of()`` from the
inode pointer, thereby allowing the netfslib helper functions to be pointed to
directly by the VFS/VM operation tables.

The structure contains the following fields that are of interest to the
filesystem:

 * ``inode``

   The VFS inode structure.

 * ``ops``

   The set of operations provided by the network filesystem to netfslib.

 * ``cache``

   Local caching cookie, or NULL if no caching is enabled.  This field does not
   exist if fscache is disabled.

 * ``remote_i_size``

   The size of the file on the server.  This differs from inode->i_size if
   local modifications have been made but not yet written back.

 * ``flags``

   A set of flags, some of which the filesystem might be interested in:

   * ``NETFS_ICTX_MODIFIED_ATTR``

     Set if netfslib modifies mtime/ctime.  The filesystem is free to ignore
     this or clear it.

   * ``NETFS_ICTX_UNBUFFERED``

     Do unbuffered I/O upon the file.  Like direct I/O but without the
     alignment limitations.  RMW will be performed if necessary.  The pagecache
     will not be used unless mmap() is also used.

   * ``NETFS_ICTX_WRITETHROUGH``

     Do writethrough caching upon the file.  I/O will be set up and dispatched
     as buffered writes are made to the page cache.  mmap() does the normal
     writeback thing.

   * ``NETFS_ICTX_SINGLE_NO_UPLOAD``

     Set if the file has a monolithic content that must be read entirely in a
     single go and must not be written back to the server, though it can be
     cached (e.g. AFS directories).

Inode Context Helper Functions

Per-inode context helper

`netfs_inode_init(ctx, ops)`는 context 기본 초기화를 수행하고 operation table pointer를 설정합니다.

`netfs_inode(inode)`는 VFS inode 구조체에서 netfs context로 cast합니다. `netfs_i_cookie(ctx)`는 inode context에 연결된 cache cookie를 반환하며 fscache가 disabled이면 NULL입니다.

------------------------------

To help deal with the per-inode context, a number helper functions are
provided.  Firstly, a function to perform basic initialisation on a context and
set the operations table pointer::

        void netfs_inode_init(struct netfs_inode *ctx,
                              const struct netfs_request_ops *ops);

then a function to cast from the VFS inode structure to the netfs context::

        struct netfs_inode *netfs_inode(struct inode *inode);

and finally, a function to get the cache cookie pointer from the context
attached to an inode (or NULL if fscache is disabled)::

        struct fscache_cookie *netfs_i_cookie(struct netfs_inode *ctx);

Inode Locking

I/O class별 i_rwsem exclusion

Netfslib는 I/O용 `i_rwsem` locking을 관리하고 더 많은 exclusion class로 사실상 확장하는 `netfs_start_io_read`, `netfs_start_io_write`, `netfs_start_io_direct`와 대응하는 `netfs_end_io_*` 함수를 제공합니다.

Buffered read는 서로 병렬 실행할 수 있고 buffered write와도 병렬입니다. 그러나 buffered write끼리는 동시에 실행할 수 없습니다.

Direct 및 unbuffered read·write는 local buffer인 page cache를 공유하지 않고 network filesystem에서는 server가 exclusion을 관리할 것으로 기대하므로 서로 병렬 실행할 수 있습니다. Ceph 등은 예외일 수 있습니다.

Truncate, fallocate 같은 주요 inode modification은 `i_rwsem`에 직접 접근해야 합니다. mmap access는 다른 모든 class와 병렬일 수 있고 intra-file loopback DIO read/write의 buffer가 될 수도 있으며 unbuffered file에서도 허용될 수 있습니다.

-------------

A number of functions are provided to manage the locking of i_rwsem for I/O and
to effectively extend it to provide more separate classes of exclusion::

        int netfs_start_io_read(struct inode *inode);
        void netfs_end_io_read(struct inode *inode);
        int netfs_start_io_write(struct inode *inode);
        void netfs_end_io_write(struct inode *inode);
        int netfs_start_io_direct(struct inode *inode);
        void netfs_end_io_direct(struct inode *inode);

The exclusion breaks down into four separate classes:

 1) Buffered reads and writes.

    Buffered reads can run concurrently each other and with buffered writes,
    but buffered writes cannot run concurrently with each other.

 2) Direct reads and writes.

    Direct (and unbuffered) reads and writes can run concurrently since they do
    not share local buffering (i.e. the pagecache) and, in a network
    filesystem, are expected to have exclusion managed on the server (though
    this may not be the case for, say, Ceph).

 3) Other major inode modifying operations (e.g. truncate, fallocate).

    These should just access i_rwsem directly.

 4) mmap().

    mmap'd accesses might operate concurrently with any of the other classes.
    They might form the buffer for an intra-file loopback DIO read/write.  They
    might be permitted on unbuffered files.

Inode Writeback

Writeback resource pin과 unpin

Netfslib는 inode가 dirty될 때 fscache cookie 같은 향후 writeback resource를 pin합니다. Cache가 active이면 folio dirty 시 `I_PINNING_NETFS_WB`를 설정하여 cache structure 폐기와 cache space culling을 막고, 이미 설정된 경우 resource를 다시 얻지 않게 합니다.

VM inode writeback 중 inode lock 안에서 flag를 clear하고 설정되어 있었다는 사실을 `struct writeback_control::unpinned_netfs_wb`로 이전합니다. 이 값이 설정되면 `write_inode` procedure를 강제합니다.

Filesystem의 `->write_inode()`가 cleanup을 수행하고 다시 netfs cleanup을 호출합니다. `netfs_unpin_writeback(inode, wbc)`가 resource unpin을 담당하며 다른 작업이 없다면 그대로 `.write_inode` method로 지정할 수 있습니다.

Inode가 삭제되면 filesystem `write_inode`가 호출되지 않을 수 있습니다. 그러므로 `->evict_inode()`에서 `clear_inode()`보다 먼저 `netfs_clear_inode_writeback(inode, aux)`를 반드시 호출해야 합니다.

---------------

Netfslib will pin resources on an inode for future writeback (such as pinning
use of an fscache cookie) when an inode is dirtied.  However, this pinning
needs careful management.  To manage the pinning, the following sequence
occurs:

 1) An inode state flag ``I_PINNING_NETFS_WB`` is set by netfslib when the
    pinning begins (when a folio is dirtied, for example) if the cache is
    active to stop the cache structures from being discarded and the cache
    space from being culled.  This also prevents re-getting of cache resources
    if the flag is already set.

 2) This flag then cleared inside the inode lock during inode writeback in the
    VM - and the fact that it was set is transferred to ``->unpinned_netfs_wb``
    in ``struct writeback_control``.

 3) If ``->unpinned_netfs_wb`` is now set, the write_inode procedure is forced.

 4) The filesystem's ``->write_inode()`` function is invoked to do the cleanup.

 5) The filesystem invokes netfs to do its cleanup.

To do the cleanup, netfslib provides a function to do the resource unpinning::

        int netfs_unpin_writeback(struct inode *inode, struct writeback_control *wbc);

If the filesystem doesn't need to do anything else, this may be set as a its
``.write_inode`` method.

Further, if an inode is deleted, the filesystem's write_inode method may not
get called, so::

        void netfs_clear_inode_writeback(struct inode *inode, const void *aux);

must be called from ``->evict_inode()`` *before* ``clear_inode()`` is called.


High-Level VFS API

Lock을 자체 관리하는 read/write_iter

Netfslib는 VFS operation을 위임할 고수준 API 묶음을 제공합니다. 내부에서 filesystem·cache를 호출해 I/O 크기를 협상하고 RPC를 issue하며 여러 개입 지점을 제공합니다.

표준 VFS `read_iter`·`write_iter`에 직접 지정할 수 있는 함수는 `netfs_file_read_iter`, `netfs_file_write_iter`, `netfs_buffered_read_iter`, `netfs_unbuffered_read_iter`, `netfs_unbuffered_write_iter`입니다.

이 함수들은 inode locking을 직접 수행합니다. 앞의 두 generic file 함수는 상황에 맞게 buffered I/O와 DIO를 전환합니다.

==================

Netfslib provides a number of sets of API calls for the filesystem to delegate
VFS operations to.  Netfslib, in turn, will call out to the filesystem and the
cache to negotiate I/O sizes, issue RPCs and provide places for it to intervene
at various times.

Unlocked Read/Write Iter
------------------------

The first API set is for the delegation of operations to netfslib when the
filesystem is called through the standard VFS read/write_iter methods::

        ssize_t netfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter);
        ssize_t netfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from);
        ssize_t netfs_buffered_read_iter(struct kiocb *iocb, struct iov_iter *iter);
        ssize_t netfs_unbuffered_read_iter(struct kiocb *iocb, struct iov_iter *iter);
        ssize_t netfs_unbuffered_write_iter(struct kiocb *iocb, struct iov_iter *from);

They can be assigned directly to ``.read_iter`` and ``.write_iter``.  They
perform the inode locking themselves and the first two will switch between
buffered I/O and DIO as appropriate.

Pre-Locked Read/Write Iter

Filesystem이 lock을 보유하는 iter API

Filesystem이 netfslib 호출 전후에 locked section 안에서 Ceph capability 협상 같은 추가 작업을 해야 할 때 pre-locked API를 사용합니다.

`netfs_unbuffered_read_iter_locked()`는 `.read_iter`에 직접 지정하면 안 되며 호출 전에 filesystem이 inode lock을 수행해야 합니다. Buffered read에는 `filemap_read()`를 사용합니다.

Write용 `netfs_buffered_write_iter_locked`, `netfs_perform_write`, `netfs_unbuffered_write_iter_locked`도 `.write_iter`에 직접 지정할 수 없고 filesystem이 먼저 inode를 lock해야 합니다.

앞의 두 함수는 buffered write입니다. 첫 함수는 표준 write check 뒤 둘째 함수로 이동하고, filesystem이 check를 직접 하면 둘째를 바로 쓸 수 있습니다. 셋째는 unbuffered 또는 DIO write입니다.

세 write 함수는 모두 writeback group pointer를 받으며 사용하지 않으면 NULL이어야 합니다. 수정되는 folio에 group을 설정하고, 다른 group이 이미 표시되어 있으면 먼저 flush합니다. Writeback API는 특정 group만 골라 writeback할 수 있습니다.

--------------------------

The second API set is for the delegation of operations to netfslib when the
filesystem is called through the standard VFS methods, but needs to do some
other stuff before or after calling netfslib whilst still inside locked section
(e.g. Ceph negotiating caps).  The unbuffered read function is::

        ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *iter);

This must not be assigned directly to ``.read_iter`` and the filesystem is
responsible for performing the inode locking before calling it.  In the case of
buffered read, the filesystem should use ``filemap_read()``.

There are three functions for writes::

        ssize_t netfs_buffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *from,
                                                 struct netfs_group *netfs_group);
        ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,
                                    struct netfs_group *netfs_group);
        ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *iter,
                                                   struct netfs_group *netfs_group);

These must not be assigned directly to ``.write_iter`` and the filesystem is
responsible for performing the inode locking before calling them.

The first two functions are for buffered writes; the first just adds some
standard write checks and jumps to the second, but if the filesystem wants to
do the checks itself, it can use the second directly.  The third function is
for unbuffered or DIO writes.

On all three write functions, there is a writeback group pointer (which should
be NULL if the filesystem doesn't use this).  Writeback groups are set on
folios when they're modified.  If a folio to-be-modified is already marked with
a different group, it is flushed first.  The writeback API allows writing back
of a specific group.

Memory-Mapped I/O API

mmap page_mkwrite와 monolithic file

`netfs_page_mkwrite(vmf, netfs_group)`는 filesystem이 `.page_mkwrite`를 netfslib에 위임하도록 합니다. 호출 전에 inode lock을 잡으면 안 됩니다. Locked write 함수처럼 writeback group을 받고 writeable로 만들 page가 다른 group이면 먼저 flush합니다.

Content를 RPC 한 번에 전부 읽고 server에는 writeback하지 않는 monolithic blob용 API도 있습니다. AFS directory가 예이며 local cache에는 저장·갱신할 수 있습니다.

`netfs_read_single`은 cache data가 있으면 우선 사용해 file을 buffer로 읽습니다. `netfs_single_mark_inode_dirty`는 이후 writeback을 유발하도록 inode를 dirty로 표시하고 `netfs_writeback_single`은 writeback code에서 data를 cache에 씁니다.

이 API를 쓰는 inode에는 `NETFS_ICTX_SINGLE_NO_UPLOAD`를 설정해야 합니다. Writeback 함수의 buffer는 `ITER_FOLIOQ` type이어야 합니다.

---------------------

An API for support of mmap()'d I/O is provided::

        vm_fault_t netfs_page_mkwrite(struct vm_fault *vmf, struct netfs_group *netfs_group);

This allows the filesystem to delegate ``.page_mkwrite`` to netfslib.  The
filesystem should not take the inode lock before calling it, but, as with the
locked write functions above, this does take a writeback group pointer.  If the
page to be made writable is in a different group, it will be flushed first.

Monolithic Files API
--------------------

There is also a special API set for files for which the content must be read in
a single RPC (and not written back) and is maintained as a monolithic blob
(e.g. an AFS directory), though it can be stored and updated in the local cache::

        ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_iter *iter);
        void netfs_single_mark_inode_dirty(struct inode *inode);
        int netfs_writeback_single(struct address_space *mapping,
                                   struct writeback_control *wbc,
                                   struct iov_iter *iter);

The first function reads from a file into the given buffer, reading from the
cache in preference if the data is cached there; the second function allows the
inode to be marked dirty, causing a later writeback; and the third function can
be called from the writeback code to write the data to the cache, if there is
one.

The inode should be marked ``NETFS_ICTX_SINGLE_NO_UPLOAD`` if this API is to be
used.  The writeback function requires the buffer to be of ITER_FOLIOQ type.

High-Level VM API

High-level VM API와 deprecated PG_private_2

VM operation을 위임하는 함수는 `netfs_readahead`, `netfs_read_folio`, `netfs_writepages`, `netfs_dirty_folio`, `netfs_invalidate_folio`, `netfs_release_folio`입니다. 모두 `address_space_operations` method이므로 operation table에 직접 설정할 수 있습니다.

아직 `->write_begin`을 쓰는 파일시스템을 위한 `netfs_write_begin`도 있지만 deprecated `PG_private_2` flag를 사용하므로 새 코드에서 사용하면 안 됩니다.

==================

Netfslib also provides a number of sets of API calls for the filesystem to
delegate VM operations to.  Again, netfslib, in turn, will call out to the
filesystem and the cache to negotiate I/O sizes, issue RPCs and provide places
for it to intervene at various times::

        void netfs_readahead(struct readahead_control *);
        int netfs_read_folio(struct file *, struct folio *);
        int netfs_writepages(struct address_space *mapping,
                             struct writeback_control *wbc);
        bool netfs_dirty_folio(struct address_space *mapping, struct folio *folio);
        void netfs_invalidate_folio(struct folio *folio, size_t offset, size_t length);
        bool netfs_release_folio(struct folio *folio, gfp_t gfp);

These are ``address_space_operations`` methods and can be set directly in the
operations table.

Deprecated PG_private_2 API
---------------------------

There is also a deprecated function for filesystems that still use the
``->write_begin`` method::

        int netfs_write_begin(struct netfs_inode *inode, struct file *file,
                              struct address_space *mapping, loff_t pos, unsigned int len,
                              struct folio **_folio, void **_fsdata);

It uses the deprecated PG_private_2 flag and so should not be used.


I/O Request API

netfs_io_request field와 request flag

`struct netfs_io_request`는 request 전체를 관리하며 filesystem을 대신해 resource·state를 보유하고 결과 collection을 추적합니다.

`origin`은 readahead, read_folio, DIO read, writeback 등 request 기원을 나타냅니다. `inode`와 `mapping`은 읽는 file의 inode와 address space이며 mapping이 `inode->i_data`일 수도 아닐 수도 있습니다.

`group`은 이 request가 다루는 writeback group 또는 NULL이며 group reference를 보유합니다. `io_streams`는 parallel subrequest stream array입니다. 현재 두 개이고 `NR_IO_STREAMS`가 array size를 나타내며 미래에 확장될 수 있습니다.

`netfs_priv`와 `netfs_priv2`는 network filesystem private data입니다. Helper 인수로 전달하거나 request 진행 중 설정할 수 있습니다.

`start`와 `len`은 read request 시작 file position과 길이이며 `->expand_readahead()`가 바꿀 수 있습니다. `i_size`는 request 시작 시 file size이고 `debug_id`는 trace line에 표시하는 operation 번호입니다.

`NETFS_RREQ_RETRYING`은 retry 생성 중 library가 설정합니다. `NETFS_RREQ_PAUSE`는 filesystem이 subrequest issue loop 일시 정지를 요청할 때 설정할 수 있지만 netfslib도 설정할 수 있어 주의해야 합니다.

`NETFS_RREQ_NONBLOCK`은 caller가 nonblocking mode를 설정했음을 library가 표시하고, filesystem은 block해야 했음을 `NETFS_RREQ_BLOCKED`로 표시할 수 있습니다.

`NETFS_RREQ_USE_PGPRIV2`는 cache write 중 folio 추적에 `PG_private_2`를 쓰려는 filesystem용이지만 해당 flag 제거 예정으로 deprecated입니다. 더 많은 private data가 필요하면 request 구조체를 감싸고 자체 allocator를 제공해야 합니다.

===============

The I/O request API comprises a number of structures and a number of functions
that the filesystem may need to use.

Request Structure
-----------------

The request structure manages the request as a whole, holding some resources
and state on behalf of the filesystem and tracking the collection of results::

        struct netfs_io_request {
                enum netfs_io_origin        origin;
                struct inode                *inode;
                struct address_space        *mapping;
                struct netfs_group        *group;
                struct netfs_io_stream        io_streams[];
                void                        *netfs_priv;
                void                        *netfs_priv2;
                unsigned long long        start;
                unsigned long long        len;
                unsigned long long        i_size;
                unsigned int                debug_id;
                unsigned long                flags;
                ...
        };

Many of the fields are for internal use, but the fields shown here are of
interest to the filesystem:

 * ``origin``

   The origin of the request (readahead, read_folio, DIO read, writeback, ...).

 * ``inode``
 * ``mapping``

   The inode and the address space of the file being read from.  The mapping
   may or may not point to inode->i_data.

 * ``group``

   The writeback group this request is dealing with or NULL.  This holds a ref
   on the group.

 * ``io_streams``

   The parallel streams of subrequests available to the request.  Currently two
   are available, but this may be made extensible in future.  ``NR_IO_STREAMS``
   indicates the size of the array.

 * ``netfs_priv``
 * ``netfs_priv2``

   The network filesystem's private data.  The value for this can be passed in
   to the helper functions or set during the request.

 * ``start``
 * ``len``

   The file position of the start of the read request and the length.  These
   may be altered by the ->expand_readahead() op.

 * ``i_size``

   The size of the file at the start of the request.

 * ``debug_id``

   A number allocated to this operation that can be displayed in trace lines
   for reference.

 * ``flags``

   Flags for managing and controlling the operation of the request.  Some of
   these may be of interest to the filesystem:

   * ``NETFS_RREQ_RETRYING``

     Netfslib sets this when generating retries.

   * ``NETFS_RREQ_PAUSE``

     The filesystem can set this to request to pause the library's subrequest
     issuing loop - but care needs to be taken as netfslib may also set it.

   * ``NETFS_RREQ_NONBLOCK``
   * ``NETFS_RREQ_BLOCKED``

     Netfslib sets the first to indicate that non-blocking mode was set by the
     caller and the filesystem can set the second to indicate that it would
     have had to block.

   * ``NETFS_RREQ_USE_PGPRIV2``

     The filesystem can set this if it wants to use PG_private_2 to track
     whether a folio is being written to the cache.  This is deprecated as
     PG_private_2 is going to go away.

If the filesystem wants more private data than is afforded by this structure,
then it should wrap it and provide its own allocator.

Stream Structure

netfs_io_stream field와 EOF 확장

Request는 destination이 서로 다른 `struct netfs_io_stream` 하나 이상으로 구성됩니다. Read request는 stream 0만 사용하며 source가 다른 subrequest를 섞을 수 있습니다. Write request는 stream 0이 server, stream 1이 cache용입니다.

Buffered writeback에서 일반 dirty folio를 만나기 전에는 stream 0을 활성화하지 않습니다. 발견 시 `->begin_writeback()`을 호출하고 filesystem이 stream을 available로 표시할 수 있습니다.

`stream_nr`는 request 안 stream 번호이고 `avail`은 사용 가능 여부입니다. `->begin_writeback()` 안에서 filesystem이 stream 0의 `avail`을 설정해야 합니다.

`sreq_max_len`과 `sreq_max_segs`는 `->prepare_read()` 또는 `->prepare_write()`가 각 subrequest의 최대 byte 수와 optional 최대 segment 수를 설정합니다. Segment 값 0은 제한이 없다는 뜻입니다.

`submit_extendable_to`는 available buffer가 허용하는 범위에서 EOF 너머로 subrequest를 round up할 수 있는 크기입니다. Cache가 EOF marker를 가로지르는 DIO read·write 가능 여부를 판단할 수 있게 합니다.

----------------

A request is comprised of one or more parallel streams and each stream may be
aimed at a different target.

For read requests, only stream 0 is used.  This can contain a mixture of
subrequests aimed at different sources.  For write requests, stream 0 is used
for the server and stream 1 is used for the cache.  For buffered writeback,
stream 0 is not enabled unless a normal dirty folio is encountered, at which
point ->begin_writeback() will be invoked and the filesystem can mark the
stream available.

The stream struct looks like::

        struct netfs_io_stream {
                unsigned char                stream_nr;
                bool                        avail;
                size_t                        sreq_max_len;
                unsigned int                sreq_max_segs;
                unsigned int                submit_extendable_to;
                ...
        };

A number of members are available for access/use by the filesystem:

 * ``stream_nr``

   The number of the stream within the request.

 * ``avail``

   True if the stream is available for use.  The filesystem should set this on
   stream zero if in ->begin_writeback().

 * ``sreq_max_len``
 * ``sreq_max_segs``

   These are set by the filesystem or the cache in ->prepare_read() or
   ->prepare_write() for each subrequest to indicate the maximum number of
   bytes and, optionally, the maximum number of segments (if not 0) that that
   subrequest can support.

 * ``submit_extendable_to``

   The size that a subrequest can be rounded up to beyond the EOF, given the
   available buffer.  This allows the cache to work out if it can do a DIO read
   or write that straddles the EOF marker.

Subrequest Structure

netfs_io_subrequest 진행·경계 flag

`struct netfs_io_subrequest`는 전체 request의 slice인 독립 I/O 단위를 관리합니다. 하나의 source만 접근하는 것이 원칙이지만 library가 source type fallback을 처리합니다.

`rreq`는 parent request pointer이고 `io_iter`는 읽을 또는 쓸 buffer slice의 I/O iterator입니다. `start`와 `len`은 slice의 시작 file position과 길이입니다.

`transferred`는 현재까지 전송한 양입니다. 각 issue에서 전송한 길이를 누적해야 하며 `len`보다 작으면 계속하기 위해 재issue할 수 있습니다.

`NETFS_SREQ_MADE_PROGRESS`는 한 byte 이상 읽거나 썼음을 filesystem이 표시합니다. `NETFS_SREQ_HIT_EOF`는 read가 EOF에 도달했음을 나타내며 `transferred`는 EOF에서 멈춰야 합니다. Library는 third-party change나 긴 DIO request 가능성 때문에 EOF folio 크기까지 subrequest를 확장할 수 있고 초과 page cache를 clear합니다.

`NETFS_SREQ_CLEAR_TAIL`은 `transferred`부터 `len`까지 남은 slice를 clear하도록 지시합니다. `HIT_EOF`와 동시에 설정하면 안 됩니다. `NETFS_SREQ_NEED_RETRY`는 retry 요청입니다.

`NETFS_SREQ_BOUNDARY`는 Ceph object 끝처럼 filesystem structure 경계에서 subrequest가 끝남을 나타냅니다. Netfslib는 이 경계를 넘어 retile하지 않습니다.

`error`는 성공 시 0, 실패 시 negative error code입니다. `debug_index`는 trace용 slice 번호이고 `stream_nr`는 소속 request stream 번호입니다.

필요하면 `netfs_get_subrequest`와 `netfs_put_subrequest`로 추가 reference를 관리하고 trace code로 이유를 표시할 수 있습니다. 제어권을 library에 돌려준 뒤 같은 subrequest가 reissue·retry될 수 있으므로 보관한 reference 사용에 주의해야 합니다.

--------------------

Individual units of I/O are managed by the subrequest structure.  These
represent slices of the overall request and run independently::

        struct netfs_io_subrequest {
                struct netfs_io_request *rreq;
                struct iov_iter                io_iter;
                unsigned long long        start;
                size_t                        len;
                size_t                        transferred;
                unsigned long                flags;
                short                        error;
                unsigned short                debug_index;
                unsigned char                stream_nr;
                ...
        };

Each subrequest is expected to access a single source, though the library will
handle falling back from one source type to another.  The members are:

 * ``rreq``

   A pointer to the read request.

 * ``io_iter``

   An I/O iterator representing a slice of the buffer to be read into or
   written from.

 * ``start``
 * ``len``

   The file position of the start of this slice of the read request and the
   length.

 * ``transferred``

   The amount of data transferred so far for this subrequest.  This should be
   added to with the length of the transfer made by this issuance of the
   subrequest.  If this is less than ``len`` then the subrequest may be
   reissued to continue.

 * ``flags``

   Flags for managing the subrequest.  There are a number of interest to the
   filesystem or cache:

   * ``NETFS_SREQ_MADE_PROGRESS``

     Set by the filesystem to indicates that at least one byte of data was read
     or written.

   * ``NETFS_SREQ_HIT_EOF``

     The filesystem should set this if a read hit the EOF on the file (in which
     case ``transferred`` should stop at the EOF).  Netfslib may expand the
     subrequest out to the size of the folio containing the EOF on the off
     chance that a third party change happened or a DIO read may have asked for
     more than is available.  The library will clear any excess pagecache.

   * ``NETFS_SREQ_CLEAR_TAIL``

     The filesystem can set this to indicate that the remainder of the slice,
     from transferred to len, should be cleared.  Do not set if HIT_EOF is set.

   * ``NETFS_SREQ_NEED_RETRY``

     The filesystem can set this to tell netfslib to retry the subrequest.

   * ``NETFS_SREQ_BOUNDARY``

     This can be set by the filesystem on a subrequest to indicate that it ends
     at a boundary with the filesystem structure (e.g. at the end of a Ceph
     object).  It tells netfslib not to retile subrequests across it.

 * ``error``

   This is for the filesystem to store result of the subrequest.  It should be
   set to 0 if successful and a negative error code otherwise.

 * ``debug_index``
 * ``stream_nr``

   A number allocated to this slice that can be displayed in trace lines for
   reference and the number of the request stream that it belongs to.

If necessary, the filesystem can get and put extra refs on the subrequest it is
given::

        void netfs_get_subrequest(struct netfs_io_subrequest *subreq,
                                  enum netfs_sreq_ref_trace what);
        void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
                                  enum netfs_sreq_ref_trace what);

using netfs trace codes to indicate the reason.  Care must be taken, however,
as once control of the subrequest is returned to netfslib, the same subrequest
can be reissued/retried.

Filesystem Methods

Request pool·초기화·readahead·prepare_read

Filesystem은 `netfs_inode`에 `struct netfs_request_ops` table을 설정합니다. 첫 두 pointer `request_pool`, `subrequest_pool`은 optional mempool입니다. 없으면 netfslib default pool을 쓰며 filesystem이 netfs 구조체를 더 큰 자체 구조체로 감싸면 자체 pool이 필요합니다. Library가 pool에서 직접 할당합니다.

Optional `init_request`, `free_request`, `free_subrequest`는 request·subrequest에 연결한 filesystem resource를 초기화하거나 정리합니다.

Optional `expand_readahead()`는 readahead request를 양방향으로 확장합니다. 이미 할당된 영역일 수 있는 초기 region은 반드시 유지해야 합니다. Local caching이 enabled이면 cache가 먼저 확장합니다.

확장은 request의 `->start`, `->len`을 변경해 알립니다. Start를 줄였다면 len을 최소한 같은 양 이상 늘려야 합니다.

Optional `prepare_read()`는 subrequest size와 RDMA 같은 iterator region 수를 제한합니다. 값을 stream 0의 `sreq_max_len`, `sreq_max_segs`에 설정합니다. 여러 server로 분할하거나 여러 read를 in-flight하기 위해 request를 자를 수 있습니다. 성공 시 0, 실패 시 error code입니다.

------------------

The filesystem sets a table of operations in ``netfs_inode`` for netfslib to
use::

        struct netfs_request_ops {
                mempool_t *request_pool;
                mempool_t *subrequest_pool;
                int (*init_request)(struct netfs_io_request *rreq, struct file *file);
                void (*free_request)(struct netfs_io_request *rreq);
                void (*free_subrequest)(struct netfs_io_subrequest *rreq);
                void (*expand_readahead)(struct netfs_io_request *rreq);
                int (*prepare_read)(struct netfs_io_subrequest *subreq);
                void (*issue_read)(struct netfs_io_subrequest *subreq);
                void (*done)(struct netfs_io_request *rreq);
                void (*update_i_size)(struct inode *inode, loff_t i_size);
                void (*post_modify)(struct inode *inode);
                void (*begin_writeback)(struct netfs_io_request *wreq);
                void (*prepare_write)(struct netfs_io_subrequest *subreq);
                void (*issue_write)(struct netfs_io_subrequest *subreq);
                void (*retry_request)(struct netfs_io_request *wreq,
                                      struct netfs_io_stream *stream);
                void (*invalidate_cache)(struct netfs_io_request *wreq);
        };

The table starts with a pair of optional pointers to memory pools from which
requests and subrequests can be allocated.  If these are not given, netfslib
has default pools that it will use instead.  If the filesystem wraps the netfs
structs in its own larger structs, then it will need to use its own pools.
Netfslib will allocate directly from the pools.

The methods defined in the table are:

 * ``init_request()``
 * ``free_request()``
 * ``free_subrequest()``

   [Optional] A filesystem may implement these to initialise or clean up any
   resources that it attaches to the request or subrequest.

 * ``expand_readahead()``

   [Optional] This is called to allow the filesystem to expand the size of a
   readahead request.  The filesystem gets to expand the request in both
   directions, though it must retain the initial region as that may represent
   an allocation already made.  If local caching is enabled, it gets to expand
   the request first.

   Expansion is communicated by changing ->start and ->len in the request
   structure.  Note that if any change is made, ->len must be increased by at
   least as much as ->start is reduced.

 * ``prepare_read()``

   [Optional] This is called to allow the filesystem to limit the size of a
   subrequest.  It may also limit the number of individual regions in iterator,
   such as required by RDMA.  This information should be set on stream zero in::

        rreq->io_streams[0].sreq_max_len
        rreq->io_streams[0].sreq_max_segs

   The filesystem can use this, for example, to chop up a request that has to
   be split across multiple servers or to put multiple reads in flight.

   Zero should be returned on success and an error code otherwise.

issue_read와 writeback 시작 hook

Required `issue_read()`는 subrequest를 server read로 dispatch합니다. `->start`, `->len`, `->transferred`가 읽을 data 범위를, `->io_iter`가 buffer를 나타냅니다.

반환값은 없습니다. 성공·실패 어느 경우든 `netfs_read_subreq_terminated()`를 호출해야 하고 먼저 `->error`, `->transferred`, `->flags`를 갱신해야 합니다. 비동기 termination도 가능합니다.

Filesystem은 folio uptodate 설정, unlock, reference drop을 처리하면 안 됩니다. 서로 다른 subrequest가 같은 folio 집합을 겹쳐 다룰 수 있어 library가 결과를 이어 붙인 뒤 처리합니다.

Optional `done()`은 read request의 모든 folio가 unlock되고 해당하면 uptodate 표시된 뒤 호출됩니다.

Optional `update_i_size()`는 write path 여러 지점에서 filesystem의 file size 인식을 갱신하라고 요청합니다. 없으면 netfslib가 `i_size`, `i_blocks`, local cache cookie를 갱신합니다. Optional `post_modify()`는 page cache write 뒤 또는 mmap page를 writeable로 허용한 뒤 호출됩니다.

Optional `begin_writeback()`은 cache-copy 표시만 있는 것이 아닌 일반 dirty page를 발견해 server write가 필요할 때 호출됩니다. 실제 write가 필요하다는 사실을 안 뒤에만 filesystem이 writeback resource를 설정할 수 있게 합니다.

 * ``issue_read()``

   [Required] Netfslib calls this to dispatch a subrequest to the server for
   reading.  In the subrequest, ->start, ->len and ->transferred indicate what
   data should be read from the server and ->io_iter indicates the buffer to be
   used.

   There is no return value; the ``netfs_read_subreq_terminated()`` function
   should be called to indicate that the subrequest completed either way.
   ->error, ->transferred and ->flags should be updated before completing.  The
   termination can be done asynchronously.

   Note: the filesystem must not deal with setting folios uptodate, unlocking
   them or dropping their refs - the library deals with this as it may have to
   stitch together the results of multiple subrequests that variously overlap
   the set of folios.

 * ``done()``

   [Optional] This is called after the folios in a read request have all been
   unlocked (and marked uptodate if applicable).

 * ``update_i_size()``

   [Optional] This is invoked by netfslib at various points during the write
   paths to ask the filesystem to update its idea of the file size.  If not
   given, netfslib will set i_size and i_blocks and update the local cache
   cookie.
   
 * ``post_modify()``

   [Optional] This is called after netfslib writes to the pagecache or when it
   allows an mmap'd page to be marked as writable.
   
 * ``begin_writeback()``

   [Optional] Netfslib calls this when processing a writeback request if it
   finds a dirty page that isn't simply marked NETFS_FOLIO_COPY_TO_CACHE,
   indicating it must be written to the server.  This allows the filesystem to
   only set up writeback resources when it knows it's going to have to perform
   a write.
   

prepare_write·issue_write·retry·cache invalidation

Optional `prepare_write()`는 subrequest 크기와 RDMA 등에 필요한 iterator region 수를 제한하고 소속 stream의 `sreq_max_len`, `sreq_max_segs`에 설정합니다. 여러 server로 분할하거나 여러 write를 in-flight하는 데 사용할 수 있습니다.

이 callback은 error를 반환할 수 없습니다. 실패하면 `netfs_prepare_write_failed()`를 호출해야 합니다.

Required `issue_write()`는 server write를 dispatch합니다. `start`, `len`, `transferred`가 data 범위이고 `io_iter`가 buffer입니다. 반환값은 없으며 완료 전 field를 갱신한 뒤 `netfs_write_subreq_terminated()`를 호출합니다. 비동기 종료도 가능합니다.

Filesystem은 관련 folio의 dirty·writeback mark를 제거하거나 reference·pin을 잡으면 안 되고 보존 책임을 netfslib에 맡깁니다.

Optional `retry_request()`는 retry cycle 시작 시 호출되어 request, 지정 stream의 subrequest, filesystem private state를 검사하고 resource를 조정·재협상하게 합니다.

Optional `invalidate_cache()`는 local cache write가 실패했을 때 netfslib가 cache data를 invalidate하고 netfs가 제공할 수 없는 최신 coherency data를 filesystem이 제공하도록 호출합니다.

 * ``prepare_write()``

   [Optional] This is called to allow the filesystem to limit the size of a
   subrequest.  It may also limit the number of individual regions in iterator,
   such as required by RDMA.  This information should be set on stream to which
   the subrequest belongs::

        rreq->io_streams[subreq->stream_nr].sreq_max_len
        rreq->io_streams[subreq->stream_nr].sreq_max_segs

   The filesystem can use this, for example, to chop up a request that has to
   be split across multiple servers or to put multiple writes in flight.

   This is not permitted to return an error.  Instead, in the event of failure,
   ``netfs_prepare_write_failed()`` must be called.

 * ``issue_write()``

   [Required] This is used to dispatch a subrequest to the server for writing.
   In the subrequest, ->start, ->len and ->transferred indicate what data
   should be written to the server and ->io_iter indicates the buffer to be
   used.

   There is no return value; the ``netfs_write_subreq_terminated()`` function
   should be called to indicate that the subrequest completed either way.
   ->error, ->transferred and ->flags should be updated before completing.  The
   termination can be done asynchronously.

   Note: the filesystem must not deal with removing the dirty or writeback
   marks on folios involved in the operation and should not take refs or pins
   on them, but should leave retention to netfslib.

 * ``retry_request()``

   [Optional] Netfslib calls this at the beginning of a retry cycle.  This
   allows the filesystem to examine the state of the request, the subrequests
   in the indicated stream and of its own data and make adjustments or
   renegotiate resources.
   
 * ``invalidate_cache()``

   [Optional] This is called by netfslib to invalidate data stored in the local
   cache in the event that writing to the local cache fails, providing updated
   coherency data that netfs can't provide.

Terminating a subrequest

Subrequest 종료와 중간 진행 통지

Subrequest 상태가 바뀌면 cache 또는 filesystem이 termination helper로 netfslib에 알립니다.

`netfs_prepare_write_failed(subreq)`는 issue 전 prepare 단계 write 실패를 동기적으로 알리며 `error` field를 먼저 갱신해야 합니다. `prepare_read()`는 단순 abort할 수 있어 error를 반환하지만 writeback failure 처리는 더 복잡합니다.

Issue된 read는 `netfs_read_subreq_terminated(subreq)`로 종료하며 `error`, `flags`, `transferred`를 미리 갱신합니다.

`netfs_write_subrequest_terminated(_op, transferred_or_error)`는 처리 byte 수 또는 negative error를 받아 write 종료를 알리고 `kiocb` completion function으로도 사용할 수 있습니다.

`netfs_read_subreq_progress(subreq)`는 read의 incremental progress를 선택적으로 알려 일부 folio를 조기에 unlock하게 하지만 subrequest를 종료하지는 않습니다. `transferred`를 먼저 갱신해야 합니다.

------------------------

When a subrequest completes, there are a number of functions that the cache or
subrequest can call to inform netfslib of the status change.  One function is
provided to terminate a write subrequest at the preparation stage and acts
synchronously:

 * ``void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq);``

   Indicate that the ->prepare_write() call failed.  The ``error`` field should
   have been updated.

Note that ->prepare_read() can return an error as a read can simply be aborted.
Dealing with writeback failure is trickier.

The other functions are used for subrequests that got as far as being issued:

 * ``void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq);``

   Tell netfslib that a read subrequest has terminated.  The ``error``,
   ``flags`` and ``transferred`` fields should have been updated.

 * ``void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error);``

   Tell netfslib that a write subrequest has terminated.  Either the amount of
   data processed or the negative error code can be passed in.  This is
   can be used as a kiocb completion function.

 * ``void netfs_read_subreq_progress(struct netfs_io_subrequest *subreq);``

   This is provided to optionally update netfslib on the incremental progress
   of a read, allowing some folios to be unlocked early and does not actually
   terminate the subrequest.  The ``transferred`` field should have been
   updated.

Local Cache API

Local cache resource와 operation table

Netfslib는 filesystem request API와 유사하지만 별도인 local cache 구현 API를 제공합니다.

`struct netfs_cache_resources`는 cache operation table `ops`, private pointer 두 개, tracing용 fscache cookie `debug_id`, invalidation counter를 보유합니다. `fscache_invalidate()`가 counter를 증가시켜 완료 뒤 cache subrequest도 invalid로 판정할 수 있게 합니다.

`struct netfs_cache_ops`는 `end_operation`, `expand_readahead`, `prepare_read`, `read`, `prepare_write_subreq`, `issue_write` callback을 제공합니다.

Cache read의 termination callback type `netfs_io_terminated_t`는 private pointer, 전송 byte 또는 error, caller context에서 확실히 종료되는지를 나타내는 `was_async` flag를 받습니다.

---------------

Netfslib provides a separate API for a local cache to implement, though it
provides some somewhat similar routines to the filesystem request API.

Firstly, the netfs_io_request object contains a place for the cache to hang its
state::

        struct netfs_cache_resources {
                const struct netfs_cache_ops        *ops;
                void                                *cache_priv;
                void                                *cache_priv2;
                unsigned int                        debug_id;
                unsigned int                        inval_counter;
        };

This contains an operations table pointer and two private pointers plus the
debug ID of the fscache cookie for tracing purposes and an invalidation counter
that is cranked by calls to ``fscache_invalidate()`` allowing cache subrequests
to be invalidated after completion.

The cache operation table looks like the following::

        struct netfs_cache_ops {
                void (*end_operation)(struct netfs_cache_resources *cres);
                void (*expand_readahead)(struct netfs_cache_resources *cres,
                                         loff_t *_start, size_t *_len, loff_t i_size);
                enum netfs_io_source (*prepare_read)(struct netfs_io_subrequest *subreq,
                                                     loff_t i_size);
                int (*read)(struct netfs_cache_resources *cres,
                            loff_t start_pos,
                            struct iov_iter *iter,
                            bool seek_data,
                            netfs_io_terminated_t term_func,
                            void *term_func_priv);
                void (*prepare_write_subreq)(struct netfs_io_subrequest *subreq);
                void (*issue_write)(struct netfs_io_subrequest *subreq);
        };

With a termination handler function pointer::

        typedef void (*netfs_io_terminated_t)(void *priv,
                                              ssize_t transferred_or_error,
                                              bool was_async);

The methods defined in the table are:

Cache operation 종료·확장·read source 선택

Required `end_operation()`은 read request 끝에서 cache resource를 정리합니다.

Optional `expand_readahead()`는 readahead 시작 시 cache granularity에 맞도록 request를 양방향 확장합니다.

Required `prepare_read()`는 다음 request slice를 설정합니다. Subrequest의 `start`, `len`이 가능한 위치와 크기를 나타내며 cache가 granularity에 맞게 길이를 줄일 수 있습니다. File size도 참고 인수로 받습니다.

반환값 `NETFS_FILL_WITH_ZEROES`, `NETFS_DOWNLOAD_FROM_SERVER`, `NETFS_READ_FROM_CACHE`, `NETFS_INVALID_READ`는 slice를 zero-fill할지, server에서 받을지, cache에서 읽을지, 현재 위치의 slicing을 포기할지 나타냅니다.

 * ``end_operation()``

   [Required] Called to clean up the resources at the end of the read request.

 * ``expand_readahead()``

   [Optional] Called at the beginning of a readahead operation to allow the
   cache to expand a request in either direction.  This allows the cache to
   size the request appropriately for the cache granularity.

 * ``prepare_read()``

   [Required] Called to configure the next slice of a request.  ->start and
   ->len in the subrequest indicate where and how big the next slice can be;
   the cache gets to reduce the length to match its granularity requirements.

   The function is passed pointers to the start and length in its parameters,
   plus the size of the file for reference, and adjusts the start and length
   appropriately.  It should return one of:

   * ``NETFS_FILL_WITH_ZEROES``
   * ``NETFS_DOWNLOAD_FROM_SERVER``
   * ``NETFS_READ_FROM_CACHE``
   * ``NETFS_INVALID_READ``

   to indicate whether the slice should just be cleared or whether it should be
   downloaded from the server or read from the cache - or whether slicing
   should be given up at the current point.

Cache read·write dispatch 계약

Required `read()`는 cache에서 읽습니다. 시작 file offset과 길이를 포함하는 destination iterator를 받고, 해당 위치부터 data를 찾아 앞으로 seek하라는 hint도 받을 수 있습니다.

Termination handler와 private data도 전달됩니다. Handler는 전송 byte 수 또는 error와 종료가 caller context에서 확실히 일어나는지를 나타내는 flag를 받아 호출되어야 합니다.

Required `prepare_write_subreq()`는 cache write subrequest 크기와 DIO/DMA 같은 iterator region 수를 제한하고 소속 stream의 `sreq_max_len`, `sreq_max_segs`에 설정합니다. Error를 반환할 수 없으며 실패하면 `netfs_prepare_write_failed()`를 호출합니다.

Required cache `issue_write()`는 subrequest를 cache write로 dispatch합니다. 범위는 `start`, `len`, `transferred`, buffer는 `io_iter`가 나타냅니다. 반환값은 없고 field를 갱신한 뒤 `netfs_write_subreq_terminated()`를 호출해야 하며 비동기 종료가 가능합니다.

 * ``read()``

   [Required] Called to read from the cache.  The start file offset is given
   along with an iterator to read to, which gives the length also.  It can be
   given a hint requesting that it seek forward from that start position for
   data.

   Also provided is a pointer to a termination handler function and private
   data to pass to that function.  The termination function should be called
   with the number of bytes transferred or an error code, plus a flag
   indicating whether the termination is definitely happening in the caller's
   context.

 * ``prepare_write_subreq()``

   [Required] This is called to allow the cache to limit the size of a
   subrequest.  It may also limit the number of individual regions in iterator,
   such as required by DIO/DMA.  This information should be set on stream to
   which the subrequest belongs::

        rreq->io_streams[subreq->stream_nr].sreq_max_len
        rreq->io_streams[subreq->stream_nr].sreq_max_segs

   The filesystem can use this, for example, to chop up a request that has to
   be split across multiple servers or to put multiple writes in flight.

   This is not permitted to return an error.  In the event of failure,
   ``netfs_prepare_write_failed()`` must be called.

 * ``issue_write()``

   [Required] This is used to dispatch a subrequest to the cache for writing.
   In the subrequest, ->start, ->len and ->transferred indicate what data
   should be written to the cache and ->io_iter indicates the buffer to be
   used.

   There is no return value; the ``netfs_write_subreq_terminated()`` function
   should be called to indicate that the subrequest completed either way.
   ->error, ->transferred and ->flags should be updated before completing.  The
   termination can be done asynchronously.


API Function Reference

Kernel-doc API reference

API function reference는 `include/linux/netfs.h`와 `fs/netfs/buffered_read.c`의 kernel-doc을 포함합니다. 선언·구현의 최신 parameter와 반환 규칙은 이 generated reference에서 확인해야 합니다.

======================

.. kernel-doc:: include/linux/netfs.h
.. kernel-doc:: fs/netfs/buffered_read.c