Signal / Event / IPC · Linux userspace / kernel ABI

epoll readiness and edge-triggered loops

Demonstrates in code that readiness is the possibility of progress on the next nonblocking I/O operation, not completion notification, and applies the EPOLLET drain rule.

Series
23 / 38
Build
cc -std=c17 -Wall -Wextra -O2 epoll_pipe.c -o epoll_pipe
Run
./epoll_pipe
Kernel
Linux 6.18.37 LTS

After EPOLLIN, must read succeed for the full requested length?

epoll uses registered files' poll callbacks and wait queues to collect state changes in an interest list. EPOLLIN means read can make at least some progress or observe EOF/error; it does not mean a complete message has arrived.

In edge-triggered mode, make the fd nonblocking and drain until EAGAIN so an edge from not-ready to ready is not missed. If code reads only part of the data and returns to the event loop, data can remain buffered without another edge arriving.

Structure diagram

Figure 1. The interest set and ready list inside eventpoll
interest rb-tree
fd 4 · IN|ETfd 7 · OUTfd 9 · IN|ONESHOT
target wait queues
pipe waitsocket write waiteventfd wait
ready list
epitem fd 4epitem fd 9empty slot
userspace events[]
data.u64=conn4data.u64=timer9

The interest set retains registration relationships; the ready list retains current delivery candidates. One fd can also be registered with several epoll instances.

Call path

Figure 2. From userspace code to observable results
epoll_ctl register interest
producer fd state changes
wake callback link to ready list
epoll_wait event batch
drain perform I/O until EAGAIN

Another thread or process can change state between readiness observation and actual I/O. Reevaluate every syscall return value.

Figure 3. Major points along the kernel-internal path
eventpoll interest rb-tree
ep_ptable_queue_proc target wait queue
ep_poll_callback ready list
epoll_wait events copy
file read recheck state

This diagram is not for memorizing function names. Follow which return value, file descriptor, memory mapping, or wait queue is passed to the next stage.

Linux 6.18.37 LTS source locations

Go beyond the glibc function to the files where the syscall implementation meets kernel objects. Each link points to the original file at the same tag.

FileFunction / structureWhat to inspect
fs/eventpoll.c do_epoll_ctl(), ep_poll_callback() Connect an interest item to the target wait queue
fs/eventpoll.c do_epoll_wait(), ep_send_events() Copy the ready list to a userspace event array
fs/pipe.c pipe_poll(), pipe_read() A target that makes readiness easy to compare with actual read results

Complete runnable example

The code below is not pseudocode with explanatory lines omitted. It is a minimal example that can be built and run as a file.

Buildcc -std=c17 -Wall -Wextra -O2 epoll_pipe.c -o epoll_pipe
01#define _GNU_SOURCE
02#include <errno.h>
03#include <fcntl.h>
04#include <stdio.h>
05#include <sys/epoll.h>
06#include <unistd.h>
07
08int main(void)
09{
10    int p[2];
11    if (pipe2(p, O_NONBLOCK | O_CLOEXEC) < 0)
12        return 1;
13    int ep = epoll_create1(EPOLL_CLOEXEC);
14    struct epoll_event add = { .events = EPOLLIN | EPOLLET, .data.fd = p[0] };
15    if (ep < 0 || epoll_ctl(ep, EPOLL_CTL_ADD, p[0], &add) < 0)
16        return 1;
17
18    if (write(p[1], "abcdef", 6) != 6)
19        return 1;
20    struct epoll_event event;
21    if (epoll_wait(ep, &event, 1, -1) != 1)
22        return 1;
23
24    char buffer[4];
25    for (;;) {
26        ssize_t n = read(p[0], buffer, sizeof(buffer));
27        if (n > 0)
28            fwrite(buffer, 1, (size_t)n, stdout);
29        else if (n < 0 && errno == EINTR)
30            continue;
31        else if (n < 0 && errno == EAGAIN)
32            break;
33        else
34            return n < 0;
35    }
36    putchar('\n');
37    close(p[0]); close(p[1]); close(ep);
38    return 0;
39}

Code notes

Source line 11O_NONBLOCK | O_CLOEXEC

Makes the EPOLLET drain loop return control to the event loop with EAGAIN instead of sleeping in its final read.

Source line 14EPOLLIN | EPOLLET

Requests an edge on state change rather than repeated level notification while readable. This flag is distinct from one-shot.

Source line 21epoll_wait(ep

Retrieves at most one ready event, but does not lock target state. event.data returns the application key supplied at registration.

Source line 24char buffer[4]

Deliberately uses a buffer smaller than 6 bytes so several reads are required.

Source line 31errno == EAGAIN

The current pipe has been fully drained. The next producer write can create a new edge.

Detailed behavior

01

Distinguish level, edge, and one-shot

Level-triggered mode may report the fd on every epoll_wait while it remains ready. Edge-triggered mode reduces events to state transitions but creates a drain obligation. EPOLLONESHOT disables the item after one event until explicitly rearmed with MOD.

When a worker pool handles the same fd, one-shot can limit concurrent consumers.

02

HUP and ERR can arrive together with data

Even when peer close produces EPOLLRDHUP/HUP, the socket receive buffer may still contain final data. Do not close immediately on HUP; keep processing until read returns 0.

ERR/HUP can be reported even when absent from the interest mask, so always handle them.

03

Fd-number reuse and event data

A closed fd number can be reused for a new file. If an asynchronous event object stores only the numeric fd, a stale event can be applied incorrectly to new connection state.

Place a connection-object pointer/index including a generation in data.u64 and define a close/reuse protocol.

Objects and lifetimes

ObjectCreation and releaseValues to inspect
eventpollCreated by epoll_create1 and released when the epoll fd closesinterest tree, ready list, wait queue
epitemConnected to a target file by EPOLL_CTL_ADD and removed by DEL/closeevent mask, user data
ready eventLinked to the ready list by a callback and delivered by epoll_waitLevel rechecks and one-shot state

Failure conditions and common misconceptions

Observed symptomLikely causesHow to verify
An EPOLLET connection stops progressingThe fd was not drained through EAGAINInspect the last read/write return value
Data is lost at EOFThe fd was closed immediately on HUPCheck for buffered data before read returns 0
The wrong connection is processedA stale event survives fd reuseCheck generation/owner and close ordering

Verify it yourself

  1. Change the read loop to perform only one read and verify that no new EPOLLET event arrives for the remaining 2 bytes.
  2. Remove EPOLLET and compare how level-triggered mode reports the remaining data again.
  3. Use socketpair with EPOLLRDHUP and record the ordering of final data sent before peer close and HUP.
Run./epoll_pipe
Tracestrace -e trace=pipe2,epoll_ctl,epoll_wait,read,write,close ./epoll_pipe

Primary sources