read(fd, buf, 4096) looks like a function call. It is not. It’s a controlled CPU-mode transition: the one legitimate doorway from your unprivileged code into the kernel, with hardware-enforced credentials checked at the threshold. That doorway has a toll — small enough to ignore for a 4 KB read, large enough that “how many times do we cross” quietly shapes the design of every fast I/O API of the past two decades.

The Doorway

On x86-64, glibc’s read wrapper loads the syscall number (0 for read) into rax, arguments into rdi/rsi/rdx, and executes the syscall instruction. That single instruction: switches the CPU to ring 0, swaps in the kernel’s stack via per-CPU state, saves the return address, and lands at one fixed, kernel-chosen entry point. Userspace doesn’t get to choose where it enters — that’s the whole security model.

text
  user: read(fd, buf, n)
    └─ syscall            ── ring 3 → ring 0, ~fixed cost
        └─ entry_SYSCALL_64: save regs, switch stack (+KPTI page-table swap)
            └─ sys_call_table[rax] → ksys_read()
                └─ fd table lookup → file->f_op->read_iter()
                    └─ copy_to_user(buf, ...)   ── SMAP-guarded copy
        ┌─ sysret          ── ring 0 → ring 3
  user: (continues)

Inside, the kernel validates everything: fd against the process’s file table, buf against the user address range (copy_to_user — the kernel never dereferences user pointers raw; SMAP even makes accidental dereference fault). This is why a syscall can’t be inlined, trusted, or skipped: the parameters are adversarial by definition.

strace ./app shows you this layer verbatim — it’s the ground truth of what a process asks of the OS, which is why it’s the first tool out of the bag for “what is this binary actually doing.”

The Bill

Measure a null-ish syscall (getppid in a loop, or lmbench’s lat_syscall) and you get ~50–100 ns on current hardware — versus ~1–2 ns for a function call. Where it goes: the mode switch itself (tens of cycles each way), register save/restore, and — since 2018 — the Meltdown/Spectre tax: KPTI swaps page tables on every entry/exit and mitigations fence speculation, which roughly doubled syscall latency overnight on affected CPUs (spectre_v2=off mitigations=off benchmarks made the point vividly; don’t run them in production).

There’s also an indirect cost that often exceeds the direct one: you return to userspace with polluted branch predictors, evicted cache lines and TLB entries. FlexSC-era research put this at up to half the total damage.

100 ns sounds free. Now multiply: a server doing 1M small packet ops/sec through per-packet syscalls burns a visible fraction of a core on crossings alone. That arithmetic — crossings × rate — explains a whole lineage of Linux APIs:

  • writev/sendmmsg/recvmmsg, pwritev2: batch N operations per crossing.
  • epoll: one syscall reports readiness of thousands of fds.
  • sendfile/splice: move data fd-to-fd without round-tripping through userspace buffers.
  • io_uring: the logical endpoint — submission and completion rings in shared memory; in polled mode, I/O with zero syscalls in steady state.

The Calls That Aren’t Calls

Some “syscalls” never cross at all. gettimeofday, clock_gettime, getcpu: the kernel maps a small region of code+data into every process — the vDSO — and glibc calls that, reading a kernel-maintained timestamp page in pure userspace. That’s why timing loops don’t show up in strace (a perennial source of “strace says it makes no syscalls!?” confusion) and why clock_gettime costs ~20 ns, not ~200. ltrace vs strace on the same binary makes the layering visible: library call ≠ system call.

The general lesson generalizes to your own APIs: if a boundary is expensive, the API evolves toward batching, shared memory, or moving the common case to the cheap side. The syscall interface just got there decades before your microservice did.

Takeaways

  • A syscall is a hardware privilege transition to one fixed entry point, with all arguments treated as hostile — ~50–100 ns before any real work.
  • Post-2018 mitigations (KPTI etc.) roughly doubled the crossing cost; the indirect cache/predictor pollution can cost as much again.
  • The cost×rate product drove epoll, sendmmsg, splice, and ultimately io_uring’s shared-ring, zero-crossing design.
  • vDSO calls (clock_gettime) are syscall-shaped but run in userspace — strace won’t see them.
  • strace for the crossing-level truth, perf stat for the cost; batch the boundary when the product gets big.