C atomic memory order and futexes | Linux System Programming

QUESTION

Does using an atomic store alone make other data safely visible?

An atomic operation prevents torn reads/writes and data races on that atomic object, but does not automatically order surrounding non-atomic data. The producer must write the payload and publish ready with a release store; the consumer must observe ready with an acquire load before the preceding payload write is guaranteed visible.

A futex sleeps on a kernel wait queue only if the userspace atomic word still equals an expected value. An uncontended lock completes without a syscall, entering wait/wake only under contention.

STRUCTURE

Structure diagram

Figure 1. A userspace atomic word and kernel futex wait queue

Producer CPU

payload stores
release store ready=1
futex wake

Shared cache line

payload[4]
atomic ready
modification order

Consumer CPU

acquire load
payload reads
CAS/spin fast path

Kernel futex bucket

key=(mm,address)
waiter list
timeout/signal/wake

Atomic state provides correctness; the kernel provides only a place to sleep under contention. FUTEX_WAIT rechecks the value immediately before sleeping.

CALL PATH

Call path

Figure 2. From userspace code to observable results

payload write ordinary memory

release store ready=1 publish

acquire load observe ready

payload read happens-before

futex wait/wake sleep when needed

Separate memory ordering from thread scheduling. Acquire/release establishes visibility ordering; a futex yields the CPU while waiting.

Figure 3. Major points along the kernel-internal path

user atomic fast path/CAS

FUTEX_WAIT recheck value

hash bucket waiter enqueue

FUTEX_WAKE matching key

scheduler task runnable

This diagram is not for memorizing function names. Follow which return value, file descriptor, memory mapping, or wait queue is passed to the next stage.

SOURCE COORDINATES

Linux 6.18.37 LTS source locations

Go beyond the glibc function to the files where the syscall implementation meets kernel objects. Each link points to the original file at the same tag.

File	Function / structure	What to inspect
kernel/futex/waitwake.c	futex_wait_setup(), futex_wait_queue()	Recheck the expected value and prevent lost wakeups
kernel/futex/waitwake.c	futex_wake()	Make waiters on the same futex key runnable
kernel/futex/core.c	get_futex_key()	Convert a private virtual address or shared backing into a key

COMPLETE PROGRAM

Complete runnable example

The code below is not pseudocode with explanatory lines omitted. It is a minimal example that can be built and run as a file.

Buildcc -std=c17 -Wall -Wextra -O2 -pthread atomic_publish.c -o atomic_publish

01#include <pthread.h>
02#include <stdatomic.h>
03#include <stdio.h>
04
05struct message {
06    int payload[4];
07    atomic_int ready;
08};
09
10static void *producer(void *argument)
11{
12    struct message *message = argument;
13    message->payload[0] = 10;
14    message->payload[1] = 20;
15    message->payload[2] = 30;
16    message->payload[3] = 40;
17    atomic_store_explicit(&message->ready, 1, memory_order_release);
18    return NULL;
19}
20
21int main(void)
22{
23    struct message message = { .payload = {0}, .ready = ATOMIC_VAR_INIT(0) };
24    pthread_t thread;
25    pthread_create(&thread, NULL, producer, &message);
26    while (atomic_load_explicit(&message.ready, memory_order_acquire) == 0)
27        ;
28    printf("%d %d %d %d\n", message.payload[0], message.payload[1],
29           message.payload[2], message.payload[3]);
30    pthread_join(thread, NULL);
31    return 0;
32}

CODE NOTES

Code notes

Source line 6int payload[4]

Although payload is non-atomic, only the producer writes it before publication and the consumer reads it after acquire, creating happens-before and avoiding a data race.

Source line 7atomic_int ready

Only the publication flag is atomic. Whether it is lock-free depends on the implementation and the type/alignment.

Source line 17memory_order_release

Prevents the preceding payload store from moving after ready=1 and publishes it to an acquire reader.

Source line 26memory_order_acquire

After reading 1, prevents the payload load from moving before the ready load and synchronizes with the release that published the same value.

Source line 26== 0)

The example busy-waits and is inefficient except for a short delay. Combine sleeping through atomic_wait/C++ or futex/condition variable.

DETAILS

Detailed behavior

relaxed is atomic but does not publish

memory_order_relaxed loads/stores preserve ready's own modification order but do not order payload accesses. They fit statistics counters and similar cases with no invariant over other data.

Do not mistake a test that happens to work on x86 for a portable guarantee from the C memory model.

Compare-exchange has success and failure orders

Successful CAS can publish/acquire lock ownership, but a failure path performs no write and therefore cannot use release order. The semantics that update expected must also be part of the loop design.

ABA, object lifetime, and reclamation are not solved by one atomic-pointer CAS.

A futex wait checks the condition again

An unlock/wake can occur after userspace sees the lock held but before the syscall begins. FUTEX_WAIT rechecks in the kernel that the current word equals expected and, if it differs, returns EAGAIN without sleeping.

The userspace atomic state machine supplies correctness; the futex is a blocking optimization.

OBJECTS

Objects and lifetimes

Object	Creation and release	Values to inspect
`atomic object`	Exists for the lifetime of the shared structure; every concurrent access must obey the memory model	modification order, alignment
`payload`	Written by the producer before release and read by the consumer after matching acquire	ownership, happens-before
`futex waiter`	Sleeps in a syscall under contention and is removed on wake/timeout/signal	key, expected value, priority

FAILURE PATH

Failure conditions and common misconceptions

Observed symptom	Likely causes	How to verify
Values break on another architecture	Relaxed publication or a data race	Use TSan and review memory orders
CPU 100%	An unbounded spin loop	Measure wait duration and switch to a futex/condition variable
missed wakeup	State changes do not match the futex expected-value protocol	Inspect the CAS/state diagram and syscall returns

LAB

Verify it yourself

Analyze a variant with release/acquire changed to relaxed on ARM hardware and under ThreadSanitizer, but do not deploy the incorrect code.
Implement adaptive waiting that switches to a condition variable after a spin count, then compare latency and CPU use.
Write raw futex WAIT/WAKE wrappers and confirm EAGAIN when the expected value has already changed.

Run./atomic_publish

Tracestrace -f -e trace=futex,clone ./atomic_publish

PRIMARY REFERENCES

Primary sources

C atomic operations futex(2)futex(7)