Harshith Sunku — Blog

An AI Coworker in an LXC Container

Sun, 12 Jul 2026 00:00:00 GMT

There’s a container on my Proxmox host — Ubuntu 25.04, 4 vCPUs, 8 GB — whose only job is to run a coding agent with access to this site’s repository. It has written posts, redesigned pages, fixed SEO bugs, and shipped the search feature you can use right now. Not as a copilot in my editor: as a coworker with its own machine.

The problem this post answers is the one I actually lost sleep over: how do you give an autonomous agent commit access and still sleep at night? I didn’t know when I started — that uncertainty is exactly why the lab got to host the experiment. The answer turned out to be the same answer as for any junior engineer: constrain the blast radius and make review cheap.

Why the agent gets its own container

The first decision was isolation. The agent runs in an LXC container rather than on my workstation, which buys exactly what containers buy: a filesystem boundary, a resource cgroup, and a snapshot story. If an agent-driven npm install goes sideways, the damage is confined to a ZFS dataset I can roll back in one command. It also makes the agent’s environment reproducible — the same reason we put CI in containers, as I covered in the container internals post.

Sizing turned out to be trivial: the model runs in a datacenter, not here, so the container is doing text processing and site builds. 4 vCPUs and 8 GB never break a sweat; the full Astro build with 90 posts and OG image generation takes about a minute. (If you want to know why local inference wasn’t an option, the memory-bandwidth math answers it — a GPU-less container with an 8 GB ceiling isn’t in that conversation, and frontier-model quality wasn’t negotiable for prose.)

API traffic doesn’t go straight to providers either. It routes through an LLM proxy on an Oracle free-tier VM, so keys live off my LAN and I get one choke point for logging and rate limits.

The pipeline is the guardrail

The agent inherits the same publishing pipeline the whole site runs on, and that pipeline is doing quiet security work:

Everything is a git commit. There is no path to production that skips version control. Every change the agent makes is diffable, attributable (commits are co-signed with the agent’s name), and revertible.
CI is the reviewer that never trusts. Type-check plus full build on every push. The content schema is strict — an invalid frontmatter field or a broken internal link pattern fails the build before deploy.
draft: true is a staging area. Posts can land in the repo without going live, so “write it” and “publish it” are separate decisions.
AGENTS.md is the onboarding doc. The repo carries its own instructions — voice, conventions, schema, do-nots. The agent reads it the way a new hire reads a runbook, and it works for the same reason runbooks work: the knowledge lives next to the system, not in someone’s head.

Notice what’s not in the list: trust in the model. The loop is designed so that a bad change is cheap to catch and cheap to undo — the same philosophy as the agent-loop architecture itself, where the model proposes and the harness disposes.

What it’s actually like

The honest report: the agent is excellent at the work that has a verifier. “Make the sitemap emit lastmod and prove it in the build output” comes back done, with the grep to show it. It’s weaker where taste is the spec — I still edit for voice, and the MCP integrations that let it check its own output (build logs, link checkers, search indexes) matter more than raw model quality.

The surprise was workflow, not capability: an agent with its own machine and a queue beats an agent in your editor, because it turns “help me type” into “here’s a task, ping me with the diff.” That’s a delegation model, and delegation needs exactly the guardrails above — which is why the pipeline came first.

Takeaways

Give an autonomous agent the same thing you’d give a new hire: an isolated machine, a documented repo, and a review process — not raw trust.
LXC + ZFS snapshots make the blast radius a rollback, not an incident.
Route agent API traffic through a proxy you control: keys off the LAN, one place to log and limit.
Make CI strict enough that “the build passed” means something; then let the agent iterate against it.
The productivity unlock is delegation with verification — the model proposes, git and CI dispose.

Who's Allowed to Talk to What: Identity and Access in the Homelab

Sun, 12 Jul 2026 00:00:00 GMT

The problem announced itself the day the third self-hosted service asked me to create its fourth local admin account. Every service wanted its own users, its own passwords, its own idea of who I am — and every one of those accounts was an orphan credential I would forget to rotate and forget to delete. Meanwhile the network side looked done: part 1 had carved everything into VLANs by trust. But segmentation answers which segments can exchange packets and quietly dodges the harder question: which people and devices are allowed to do what. A VLAN doesn’t know who you are. If your security model ends at segmentation, anyone with a foothold in the right subnet inherits everything that subnet can reach — perimeter thinking, and it fails the same way at home as it does in companies. All shell, no bones.

What I tried first was what everyone tries first: keep minting local accounts and keep a mental map of them. That scales to about three services. The fix the lab taught me is the same one enterprises pay real money for — identity as its own layer, with one place accounts are born and one place they die.

Identity gets its own machine — and its own segment

User accounts, groups, and authentication now live in a dedicated identity appliance. It’s one of the few VMs rather than containers in the lab, and it sits in its own VLAN, because the identity service is the root of trust: compromise it and every downstream permission is fiction. The isolation calculus that felt like overkill for a media server is the bare minimum here — a full kernel boundary, a segment nothing else shares, and the management plane unreachable from anything untrusted.

Centralizing identity sounds like enterprise ceremony for a house, right up until you’ve lived the local-admin-account treadmill. One directory, one place accounts are born, one place they die.

Tailscale is the front door

Remote access is where identity and networking actually meet. Nothing in the lab is port-forwarded; the way in is Tailscale — WireGuard underneath, but the part that matters here is who a connection is, not just where it’s from:

Nodes authenticate as identities, not addresses. A device joins the mesh by logging in; access follows the login. Lose a laptop and you revoke the device, not re-key a VPN and re-distribute configs.
ACLs are policy as code. “This user’s devices may reach the hypervisor’s management interface; that tagged node may reach only the reverse proxy.” It’s a reviewable text file, not a pile of firewall rules re-derived from memory.
Keys expire. Access defaults to decaying unless renewed, which is the correct default for anything you’ll otherwise forget about.

The mental shift: the VPN isn’t a tunnel into a trusted network — it’s a per-device, per-identity grant. Being “on the network” stops being a permission by itself.

recovery must not depend on the thing being recovered

Public exposure follows the same scoping logic. The Cloudflare Tunnel from part 1 publishes specific services, not a network: an outbound connection carries named hostnames to a container behind the reverse proxy, and the blast radius of “public” ends at those hostnames. Public ≠ LAN. (This blog dodges the question entirely — it’s static files on a CDN and touches none of this.)

Design for the day identity is down

Here’s the failure I almost built for myself: the identity VM goes down, and every login in the lab depends on it — including, if you’re careless, the login you’d use to fix the identity VM. A circular dependency, waiting to be discovered at the worst time. This is the same lesson the wake-on-LAN gateway taught me at the power layer: recovery paths must not depend on the thing being recovered.

So the lab keeps deliberate break-glass paths: a local root credential on the hypervisor that lives in a password manager and nowhere on the network, and console access that works when SSO doesn’t. The test that matters isn’t “can I log in” — it’s “can I log in while the directory is off.” I ran that drill on purpose before I needed it, and I recommend the experience: a lock you can’t open in a power cut isn’t security, it’s a lockout.

Takeaways

Segmentation controls which networks talk; identity controls which people and devices do. You need both — a VLAN doesn’t know who you are.
Centralize accounts in one directory, isolate it like the root of trust it is (own VM, own segment), and stop minting local admins.
Prefer identity-based mesh access (Tailscale ACLs, expiring keys, per-device revocation) over network-based trust. “On the VPN” should not mean “allowed.”
Expose services, never networks: scope public tunnels to named hostnames.
Keep a tested break-glass path that works with identity down — the recovery rule from part 4, applied to logins. Drill it before you need it.

The Fleet: Six Machines, and What the Hypervisor Can't Do

Sun, 12 Jul 2026 00:00:00 GMT

The lab was supposed to be one machine. Part 1 made the consolidation case — one Proxmox box, twenty containers, everything snapshotted and segmented. Then I built a performance-sampling tool, asked the obvious question — does it behave the same on ARM? — and discovered that the machine I had consolidated everything onto couldn’t answer it.

This post is about the fleet that grew around that lesson. Six machines, and none of them is here for fun: each one exists because I hit a problem the hypervisor could not solve, and each one taught me something I wouldn’t have learned from a VM. That’s the whole point of this lab — it’s a classroom that happens to run services.

The center, and one cheap trick worth stealing

The Proxmox host from part 1 is still the center of gravity: an Ultra 9 285K, 96 GB of DDR5-6600, and three 2 TB Gen4 NVMe drives carrying separate ZFS pools, plus an 8 TB mirrored pool on spinning disks for bulk data.

The HDD pool taught me the first lesson. Spinning disks stream large files fine, but metadata is where they die — walking a directory tree of a few hundred thousand files means a seek per lookup, and ls starts to feel like an I/O benchmark. The fix was a ZFS special vdev: a mirrored pair of 256 GB SATA SSDs attached to the HDD pool that holds its metadata (and optionally small blocks). Directory walks hit flash; big sequential reads still stream off the platters. Two leftover SSDs turned out to be the cheapest upgrade a spinning pool can get. One caveat that matters: the special vdev is the pool — lose it and the pool is gone — which is why it’s a mirror and not a single drive.

The counters that wouldn’t count

The perf tool is where consolidation broke. Sampling profilers lean on the PMU — the CPU’s performance monitoring unit, the hardware counters behind perf stat’s cycles and cache-misses. And a hypervisor is exactly the wrong place to learn how PMUs behave: virtualized counters are incomplete when they’re exposed at all, the hypervisor multiplexes them behind your back, and the emulated ARM VM from part 1 has no real PMU whatsoever — QEMU proves your code runs on ARM, not how it performs.

So the tool got tested where counters tell the truth — on bare metal, across three genuinely different microarchitectures:

i5-3330 — Ivy Bridge, Intel’s PMU, 2012 vintage. An old baseline is a feature: if the tool works here, it isn’t accidentally depending on anything modern.
A10-9600P — AMD’s PMU, which is not Intel’s PMU. Same perf API, different counters, different quirks. You don’t learn that from documentation; you learn it when the same code returns different shapes of data.
Snapdragon 845 — the ARM PMU, big.LITTLE and all, in a phone (more on that below).

That test matrix is the lab’s thesis in miniature: the hypervisor virtualizes CPUs well and performance counters badly. Anything that touches hardware this directly needs real silicon.

A phone is the cheapest ARM dev board

The ARM machine in that matrix is a Poco F1 — Snapdragon 845, 8 GB of RAM, 256 GB of storage — running a NetHunter kernel. NetHunter is security tooling, but that’s not why it’s here: the point is a rooted device with a replaceable kernel. That combination turns a retired phone into an ARM development board with better specs than most actual dev boards — four Cortex-A75s and four A55s of real big.LITTLE scheduling, a real ARM PMU, and a battery-backed power supply built in.

Every “works on my machine” assumption a career of x86 quietly builds up gets audited the first time your tool runs on this thing.

The bench that has real ports

Firmware work was the second thing the hypervisor lost. Flashing embedded boards — like the ESP32 wake gateway — means serial consoles and USB devices that reset and re-enumerate mid-operation. USB passthrough into a VM can be forced, but a device that drops and reattaches during a flash is a great way to end up with half-written firmware, and the failure is always ambiguous: the board, the cable, or the passthrough?

The answer is a machine nobody would miss: an i5-3330 desktop with 16 GB and an assortment of leftover disks. Real USB controllers, a real serial port, no virtualization layer to suspect. When a flash fails there, the board is actually broken — and removing a layer of doubt is worth more than the electricity it burns. It doubles as the Ivy Bridge test target above.

The machines that run the machines

The rest of the fleet is about operating the lab rather than being experimented on. An i5-1240P laptop — 32 GB, a 2 TB and a 512 GB Gen4 NVMe, Windows 11 — is the daily driver and control plane: every console, dashboard, and SSH session in the lab happens from here. And a MacBook Air M4 with 24 GB is the portable half — on the Tailscale mesh from part 1, so “at the lab” is a network property, not a physical one. The A10 laptop rounds things out as the sacrificial box: risky OS installs and one-off servers land there first, where a bad day costs nothing.

The deliberate redundancy: the control plane runs Windows, the portable runs macOS on ARM, the lab runs Linux. Three operating systems and two ISAs in daily rotation means cross-platform assumptions get caught by breakfast.

Takeaways

Consolidation is right up until the work touches hardware. PMUs, serial ports, USB quirks — that’s where bare metal earns its keep.
A hypervisor virtualizes CPUs well and performance counters badly. Test anything perf-shaped on real silicon, across more than one vendor’s PMU.
A rooted phone with a replaceable kernel is the cheapest real-ARM test device you can own — better specs than a dev board, battery included.
A mirrored special vdev is the cheapest upgrade a ZFS spinning pool can get: metadata on flash, bulk data on platters. Mirror it — it’s pool-critical.
Keep one machine nobody would miss. Risky experiments need a home where failure is free.
Next in this series: the strangest tenant on the always-on host — the AI coworker in an LXC container.

Waking the Lab: an ESP32, MQTT, and 102 Bytes

Sun, 12 Jul 2026 00:00:00 GMT

The Proxmox host in my lab runs 24/7. The machines around it don’t: the workstation and the bench boxes sleep when idle, and Wake-on-LAN is a layer-2 mechanism, so waking one remotely means something inside the LAN has to send the packet. My first wake path was the obvious one — a container on the hypervisor that publishes magic packets — and it worked right up until the day the machine I needed to wake was the hypervisor, down for a kernel upgrade. A recovery path that depends on the thing being recovered isn’t a recovery path. (Port-forwarding UDP from the internet is the other classic answer, and it violates the rule from the homelab networking post: VPN in, don’t port-forward.)

So the wake path is its own device — independent of every machine it might need to rescue, drawing about half a watt, costing less than lunch: an ESP32 that holds one MQTT subscription and broadcasts magic packets on command.

The shape of the thing

The ESP32 runs ESP-IDF — FreeRTOS underneath, lwIP for the network stack. The application is honestly small: connect to Wi-Fi, keep a session to the broker, and on a message to wol/wake/<host>, assemble the 102-byte magic packet (6×0xFF + the target MAC sixteen times) and fire it at the broadcast address on UDP port 9. The packet format and why the NIC’s pattern-matcher can parse it while the CPU is off is the previous post’s territory; this post is about what it takes to make the sender dependable.

One wrinkle a segmented network adds: broadcasts stop at VLAN boundaries. Small embedded gadgets normally belong on the quarantined IoT segment, but a magic packet sent there dies at the VLAN edge — the wake gateway needs layer-2 adjacency with its targets. In a network carved up by trust, that placement is a deliberate, documented exception — the kind of thing you decide once, on purpose, instead of discovering at 2 AM.

MQTT choices that matter

The protocol details are in the MQTT internals post; here’s how they cash out in a real device:

QoS 1, not 0, not 2. A wake command that evaporates is a failed feature — QoS 0 is out. Duplicate delivery, on the other hand, is harmless: waking an awake machine is a no-op. That’s precisely the QoS 1 contract (at least once), and it’s cheaper than QoS 2’s four-way handshake. Idempotent command + QoS 1 is the sweet spot.

No retained messages on the command topic. This is the classic trap. Retain a wake command and every reconnect replays it — the ESP32 reboots, resubscribes, and helpfully wakes the server you just shut down. Commands are events, not state; retained messages are for state.

Last Will as a health signal. The broker publishes wol/status/gateway = offline if the ESP32’s session dies. My monitoring watches that topic — a dead wake-gateway discovered at 2 AM when you need it is the worst possible discovery time.

The watchdog is the design

The failure mode of an always-on embedded device isn’t crashing — it’s wedging: Wi-Fi associated, TCP session zombied, task blocked on a semaphore that will never post. A crash reboots; a wedge lasts until you physically walk over, which defeats the entire point of a remote wake device.

So the design assumption is “this will wedge eventually,” and the answer is the ESP32’s hardware task watchdog. The main loop must check in every few seconds; the check-in is gated on actual liveness — broker connection up, subscription alive — not just “the loop is spinning.” If the session zombies, the check-in stops, the watchdog fires, and the device is back in a known-good state in about two seconds. Nobody walks anywhere.

That inversion — design for recovery, not for uptime — is the one idea from this build that generalizes to every unattended device.

Takeaways

WoL’s layer-2 constraint means something must live on the LAN; make it the cheapest, lowest-power thing that can hold an MQTT session — and keep the wake path independent of the machines it recovers.
VLAN segmentation and broadcast-based protocols fight: place the wake gateway with L2 adjacency to its targets, as a deliberate exception.
Idempotent commands + QoS 1 give you reliability without QoS 2’s overhead. Never retain command messages — retain state, publish events.
Use MQTT Last Will to monitor the monitor: a wake gateway that fails silently fails at the worst time.
For unattended devices, gate the watchdog check-in on end-to-end liveness and let the reset be your recovery path.
This device backstops the lab whose always-on host runs the AI coworker that maintains the site you’re reading. Every layer has a recovery story — next up, the same rule applied to logins.

My Homelab: One Proxmox Box, ZFS Pools, and a Properly Segmented Network

Fri, 10 Jul 2026 00:00:00 GMT

The lab didn’t start as a lab. It started as services accreting onto whatever machine happened to be on — a media server here, a script there, everything sharing one filesystem and one fate. The failure that changed it was boring and total: one bad upgrade, and everything went down together, with no isolation between the thing I broke and the things I cared about, and no undo. I rebuilt from scratch, twice.

That’s the problem this lab solves. Not “host services” — break things without breaking everything, and always have a way back. The solution became one Proxmox VE host carrying about twenty LXC containers and a handful of VMs, behind a UniFi network that does the segmentation work. Every post on this blog is now written, built, and pushed from a container in that lab. No seven-node cluster blinking in a rack — the interesting lessons are in how much one box can carry, and where the isolation boundaries actually earn their cost.

zero port-forwards

The host, and the undo button

The box is built around an Intel Core Ultra 9 285K — 24 threads on one socket — with 96 GB of RAM, running Proxmox VE 9.2 on kernel 7.0.14-4-pve. Proxmox gives me three primitives behind one API: full VMs (QEMU/KVM), system containers (LXC), and ZFS storage. At a typical moment the host sits under 1% CPU with ~18% of memory committed — the first surprise the lab taught me is that the constraint on a homelab server is almost never compute. It’s RAM and storage layout.

Storage is where the two rebuilds paid off. Everything is ZFS, split by what the disks are good at: a fast NVMe pool for container roots and VM disks, a big spinning pool (~12 TiB raw between them) for bulk data, and a separate pool that exists only to receive backups. Each container’s root is its own dataset (the blog’s workspace lives on a subvol under the fast pool), which buys the three things I actually use daily:

Snapshots before anything risky. Instantaneous, and rollback is one command. The “no undo” failure that started this lab can’t happen anymore — and it has saved me twice since.
Per-dataset quotas instead of pre-carved disk images — the pool is shared and containers take what they’re allotted.
Checksums on every block. ZFS detects the bit-rot that a plain RAID mirror silently propagates — the distinction I unpacked in RAID: what it does and doesn’t.

The VM/container line is a sizing lesson

I wrote about what a container really is — namespaces plus cgroups, no guest kernel. What the lab taught me is that this definition becomes a budget. My first instinct was VMs for everything, because VMs feel like “proper” isolation. Then I did the arithmetic: a VM pins its allocation — give it 8 GB and that memory is spoken for, plus a whole guest kernel. Twenty services as VMs would have blown through 96 GB before doing any work. An LXC container shares the host kernel; its “8 GB” is a cgroup ceiling, and unused memory stays in the host page cache where it does something useful.

So almost everything is LXC: a reverse proxy, the Cloudflare Tunnel connector, a Tailscale node, a photo library, a vector database, an LLM chat UI, a code-server, a Docker host for compose-shaped things, some monitoring, and the container that runs the coding agent maintaining this site — 4 vCPUs and 8 GB, strange enough to deserve its own post.

The VMs are the exceptions that prove the rule, each needing its own kernel: a Windows 11 guest, an identity/auth appliance I want strongly isolated, and an emulated ARM machine for cross-architecture testing. The tell from inside a container: systemd-detect-virt says lxc, and uname -r shows the host’s -pve kernel. One kernel, many userspaces.

The network does the isolation

The second lesson came from the network side, and it arrived the way these lessons do — as a “wait, that can see this?” moment. A flat network means the cheapest, least-trustworthy device on it (a smart plug, a guest’s phone) sits one hop from the hypervisor’s management interface. Compute isolation is worthless if the network hands out adjacency for free.

So the other half of the lab is a UniFi stack — a UCG Ultra gateway, a PoE switch, and a WiFi 6 access point — with the flat-network problem solved the way the homelab networking post prescribes: segmentation by trust. The management network hangs off 192.168.0.0/24, and everything else lives in its own VLAN and 10.10.x.0/24 — IoT gadgets, guest WiFi, office gear, public-facing servers, NAS traffic, VPN clients, an experiment sandbox for things I don’t trust yet, and an identity/auth segment. A dozen VLANs sounds like enterprise cosplay until the first compromised smart plug can’t see your hypervisor. Then it sounds like the minimum.

SSIDs map onto the same segments — the IoT network is 2.4 GHz-only, which is where small embedded things like my wake-on-LAN gateway live.

Nothing is port-forwarded. Remote access is Tailscale (mesh VPN in, zero inbound holes), and the few deliberately public services go out through a Cloudflare Tunnel — an outbound connection from a container, so the LAN never accepts unsolicited traffic. The blog itself doesn’t touch any of this: it’s static files on Cloudflare’s edge, because a residential connection is exactly where a public website should not live.

There’s also one machine outside the house: an Oracle Cloud free-tier VM running an LLM API proxy — keys stay off the LAN, and it survives local power cuts.

Takeaways

The lab exists to learn on. Its real product isn’t uptime — it’s the lessons: isolation boundaries, sizing budgets, and recovery paths you only understand after breaking them.
One well-specced Proxmox box covers a remarkable amount of homelab; RAM and storage tiering are the real constraints, not CPU.
Prefer LXC over VMs for trusted Linux services — shared-kernel density is the whole game on a single host. Reserve VMs for foreign kernels and hard isolation boundaries.
ZFS snapshots before every risky change; quota datasets instead of fixed images; a dedicated backup pool that does nothing else.
Segment by trust: VLANs per role, IoT quarantined, management network unreachable from anything untrusted. VPN in, tunnel out, forward nothing.
Next in this series: the fleet of bare-metal machines around this box — and what a hypervisor can’t do.

How This Blog Runs for $0 a Month

Wed, 08 Jul 2026 00:00:00 GMT

People assume a blog with full-text search, per-post social cards, RSS, and a hundred-ish pages needs a server somewhere. This one has none. The entire running cost is the domain registration — roughly the price of two coffees a year — and everything else is $0. Not “free tier that becomes $40/month when you get traffic” free. Actually free, by construction.

The trick isn’t a discount. It’s an architectural decision: nothing happens at request time. If no code runs when you load a page, there’s nothing to bill.

The pipeline

The whole system is one directed graph, and every arrow fires at build time:

nothing happens at request time

Posts are Markdown files in src/content/blog/ — one file per URL. There is no CMS and no database; the git history is the edit history, and a git revert is my rollback story. Astro reads the collection at build time, type-checks the frontmatter against a schema, and emits plain HTML.

Hosting: Cloudflare Pages

Cloudflare Pages watches the GitHub repo. Every push to main triggers a build (astro build, about a minute) and atomically swaps the deployment. The free plan serves static assets with unmetered bandwidth from their edge — the same CDN story I dug into in the HTTP caching contract, except here every response is cacheable forever because nothing is dynamic. TLS is included; the handshake terminates at their edge, not on any machine I pay for.

The limits that matter on the free plan: 500 builds per month (I use maybe 30) and a 25 MiB per-file cap (largest file here is a font subset, ~50 KiB). Traffic spikes are Cloudflare’s problem, which is exactly where I want that problem to live.

Search without a server

Full-text search is the feature everyone assumes needs a backend. It doesn’t. Pagefind runs after the build, crawls dist/, and writes a chunked, compressed index as static files:

"build": "astro build && pagefind --site dist"

The browser lazy-loads only the index fragments a query touches — a search costs a few tens of kilobytes, not a round-trip to a search service. The index is scoped with data-pagefind-body so navigation chrome doesn’t pollute results. Total infrastructure: zero.

OG images. Every post gets a 1200×630 social card at /og/<slug>.png, rendered during the build by an Astro endpoint that feeds an SVG template through sharp. When a post’s title changes, the next build regenerates its card. No image service, no on-demand rendering λ.

Fonts. The three typefaces are self-hosted woff2 files with @font-face and preload hints — no Google Fonts request, no third party in the critical path, and one less DNS lookup (a cost I’ve measured before).

CI. GitHub Actions runs tsc --noEmit and a full build on every push. Free for public repos. It’s the reviewer that never gets tired: a broken frontmatter field or a dead import fails the check before Cloudflare ever sees the commit.

The bill

Component	Provider	Cost
Hosting + CDN + TLS	Cloudflare Pages	$0
Search	Pagefind (static)	$0
Social cards	sharp at build time	$0
CI	GitHub Actions	$0
Analytics	none (deliberately)	$0
Domain	registrar	~$10/yr

The domain is the only line item, and it’s the one thing that can’t be static.

What this architecture costs you instead

Honesty section: you pay in constraints, not dollars. No comments without a third-party embed. No personalization, ever. Content changes require a build (~90 seconds from push to live), so this is the wrong shape for anything real-time. And build-time work scales with content — at thousands of posts, the OG-image step would dominate and I’d need to cache between builds.

For a technical blog, none of those bite. The failure mode of a $6 VPS is a 3 AM page; the failure mode of this stack is that I wait a minute for a deploy.

Takeaways

Move all computation to build time. If nothing runs at request time, hosting is free and outages need Cloudflare-sized incompetence.
Full-text search does not require a server — Pagefind ships a static, lazily-loaded index that costs kilobytes per query.
Social cards, fonts, sitemaps, RSS: all derivable from content at build time. Derive them; don’t serve them.
Git as CMS gives you review, rollback, and history for free — and lets CI reject broken content before it deploys.
The next post in this series covers where the rest of my infrastructure lives: the homelab.

How LLM Inference Actually Runs: Tokens, KV Cache, and Why It's Memory-Bound

Sun, 05 Jul 2026 00:00:00 GMT

I came to LLMs from the systems side, and the framing that finally made them legible wasn’t the math of attention — it was treating inference as what it is: a memory-bound serving workload with an unusual cache. Once you see that a model generating text is bandwidth-limited, not compute-limited, the whole landscape of confusing behavior (why the first token is slow but the rest are steady, why context length costs so much memory, why batching is everything) falls into place. This is that systems-eye view.

Generation Is a Loop, and Each Step Re-Reads the Model

An autoregressive LLM produces one token at a time (tokens are sub-word chunks — a tokenizer splits text into ~4-characters-ish pieces from a fixed vocabulary). Each step: feed the whole sequence so far through the network, get a probability distribution over the vocabulary, sample one token, append it, repeat.

  "The capital of France is" → [forward pass] → "Paris" (prob 0.8...)
  "The capital of France is Paris" → [forward pass] → "." → ...

The systems-critical fact: every single token requires reading all of the model’s weights. A 70-billion-parameter model at 16-bit is 140 GB; at 4-bit quantization ~35 GB. To generate one token, the hardware streams those tens of gigabytes from GPU memory through the compute units. To generate 500 tokens, it does that 500 times. The arithmetic per token is modest; the data movement is enormous — which means inference speed is gated by memory bandwidth, not FLOPs. A GPU rated for petaflops sits mostly idle during single-stream decode, waiting on memory. This one fact is why an H100’s headline compute number tells you almost nothing about its tokens-per-second.

The KV Cache: The Optimization Everything Depends On

Naively, re-processing the entire growing sequence each step is O(n²) — token 500 reprocesses 499 predecessors. The fix that makes inference tractable is the KV cache: attention computes a “key” and “value” vector per token, and those don’t change once computed. So cache them; each new token computes its own K/V and attends over the cached ones, turning per-step work from “reprocess everything” to “process one new token against the cache.”

This splits inference into two phases with completely different performance characteristics — the distinction behind every latency number you’ll see quoted:

Prefill: process the entire prompt at once, building the KV cache. Parallel across all prompt tokens, compute-heavy, fast per token. This is your time-to-first-token, and it scales with prompt length.
Decode: generate tokens one at a time, each reading the whole model + the whole KV cache. Sequential, memory-bound, one token per forward pass. This sets your tokens-per-second.

“Long prompts are slow to start, then generate steadily” is exactly this: a big prefill, then bandwidth-limited decode.

The KV cache also explains the context-length memory wall. The cache grows linearly with sequence length and is often the dominant memory consumer at long context — potentially many gigabytes for one long conversation, per request. This is why long context is expensive, why serving frameworks obsess over it, and why PagedAttention (the idea behind vLLM, ~2023) mattered so much: it manages the KV cache in fixed pages like an OS manages virtual memory, ending the fragmentation that used to waste most of it — a page table for attention, which is a deeply satisfying convergence for anyone who’s read this blog’s OS posts.

Why Serving Looks the Way It Does

Every production inference optimization is an answer to “we’re memory-bandwidth-bound, so amortize the weight reads”:

Batching is the core lever. Reading the weights to serve one request costs the same as reading them to serve fifty concurrent requests — the weights stream once, the matmul does more work per byte read. So throughput scales dramatically with batch size until compute or KV-cache memory becomes the limit. Continuous batching (swap finished sequences out and new ones in each step, rather than waiting for a whole batch to finish) is why modern servers hit high utilization. This is also the fundamental tension: latency (small batch, dedicated bandwidth) vs throughput/cost (big batch).
Quantization (16-bit → 8/4-bit weights) directly attacks the bottleneck: fewer bytes per weight = less to stream = faster decode and less memory, at some accuracy cost. It’s a bandwidth optimization first, a memory one second.
Speculative decoding breaks the one-token-per-pass sequential floor: a small cheap model drafts several tokens, the big model verifies them in one parallel pass, accepting the run that matches. Multiple tokens per expensive forward pass when the draft is good.

None of this requires understanding transformers deeply — it requires understanding that you’re moving a lot of memory repeatedly, which is the most systems-engineering problem imaginable. Profiling LLM serving is bandwidth accounting, queueing, and cache management: familiar tools pointed at a new workload.

Takeaways

LLMs decode one token at a time, re-reading all model weights per token — inference is memory-bandwidth-bound, so GPU FLOP ratings mislead.
The KV cache turns O(n²) decode into per-token work by caching unchanging key/value vectors; prefill (TTFT) and decode (tok/s) are separate performance regimes.
KV cache grows with context length and is the long-context memory wall; PagedAttention manages it like OS virtual memory.
Batching amortizes the fixed weight-read cost across requests — continuous batching is why servers stay utilized; latency vs throughput is the central trade.
Quantization cuts bytes-per-weight (bandwidth), speculative decoding cuts passes-per-token — both attack the real bottleneck.

Reading Flame Graphs: Where Your CPU Time Actually Goes

Wed, 01 Jul 2026 00:00:00 GMT

“I trace, profile, and prove it” is the tagline of this whole site, and the flame graph is the single most efficient tool for the prove it part. I lean on it enough that I built a profiler around live flame graphs streamed to a browser. But the tool only helps if you can read it correctly — and the common misreadings (mistaking depth for cost, missing that the profiler only sees on-CPU time) send people optimizing the wrong function. Here’s how sampling works and how to read the picture in ten seconds flat.

Sampling: Statistics, Not Surveillance

A sampling profiler like perf doesn’t trace every function — that would be ruinously slow and would distort the thing it measures. Instead it interrupts the program at a fixed frequency (say 99 Hz — 99, not 100, to avoid lockstep with periodic timers) and records the call stack at that instant:

  99 times/sec:  freeze → walk the stack → log it → resume
  after 30s:  ~3000 stacks, each a snapshot of "what was running"

The statistical bet is simple and sound: a function that appears in 50% of samples was on-CPU ~50% of the time. You don’t need to catch every call; you need enough samples that the proportions are trustworthy — which is why short profiles of rare events are unreliable and why 99 Hz over 30 seconds beats 4000 Hz over one. Low overhead (a few percent) is the whole reason sampling beats instrumentation for production profiling.

Reading the Graph in Ten Seconds

Thousands of stacks are unreadable as a list. The flame graph (Brendan Gregg’s invention, ~2011) aggregates them: identical stacks merge, and the result is drawn as stacked boxes.

   ┌──────────────────────────────────────────────┐  ← full profile
   │ main                                          │
   ├───────────────────────┬──────────────────────┤
   │ handle_request        │ background_flush      │
   ├───────────┬───────────┼──────────────────────┤
   │ parse     │ db_query ▓▓▓▓▓▓ (wide = HOT)      │
   └───────────┴───────────┴──────────────────────┘

The two rules that are the entire skill:

Width = time. A box’s width is the fraction of samples containing that frame. Wide boxes are where the CPU time goes. This is the only thing you’re hunting for.
Y-axis = stack depth, NOT importance. Boxes stack because one function called another. A tall tower is just deep call nesting; it says nothing about cost. The #1 beginner error is chasing tall spires — look for wide plateaus, especially wide boxes near the top (leaf functions doing actual work, not just calling down).

That’s it: scan across the top for the widest boxes, and those are your targets. Colors are usually just for visual separation (warm random palette); ignore them unless the graph specifically uses color to encode something (some tools tint by language or by whether frames are kernel vs user). Interactive versions let you click a box to zoom into that subtree — the fast path to “what is db_query spending its time on.”

The Traps That Mislead

A flame graph is honest about what it measures and silent about what it doesn’t — and the gaps are exactly where people get fooled:

It only shows on-CPU time. A program blocked on disk, network, a lock, or sleep is not running, so it generates no samples — time spent waiting is invisible on a standard CPU flame graph. If your service is slow but the flame graph looks idle or unremarkable, your bottleneck is off-CPU (I/O or lock contention), and you need an off-CPU flame graph (from scheduler tracing) instead. This single misunderstanding accounts for most “the profiler says nothing’s wrong but it’s slow” confusion.
Inlining hides functions. At -O2 the compiler inlines aggressively (see my optimization post), so a hot function may not appear as its own frame — its time is attributed to the caller it was folded into. Confusing until you remember the optimizer erased the boundary.
Broken stacks need frame pointers. Reliable stack walking wants frame pointers, which optimized builds often omit (-fno-omit-frame-pointer to keep them), or DWARF/LBR-based unwinding as the fallback. Symptoms are truncated towers or [unknown] frames — a collection problem, not a code problem. (Fedora and others flipped frame pointers back on by default in 2023 largely to make production profiling trustworthy again.)
Sampling misses the rare. A function called once for 5 ms in a 30 s profile may not appear at all. Flame graphs answer “where does the bulk go,” not “did this specific thing happen.”

Used with those caveats in mind, it’s the fastest path from “the system is slow” to a specific function and line — capture with perf record -F 99 -g -- ./app, fold, and render, or use any of the tools that do it live. The discipline is always the same: measure, read width not height, and check whether your time is even on-CPU before you trust the picture.

Takeaways

Sampling profilers snapshot the stack at fixed frequency; frame frequency ≈ time spent — low overhead, statistically sound over enough samples.
Read width, not height: wide boxes (especially wide leaves at the top) are the cost; tall towers are just deep call chains.
Standard flame graphs show only on-CPU time — I/O, lock, and sleep waits are invisible; use off-CPU graphs when the CPU view looks innocent but the service is slow.
Inlining folds hot functions into callers; missing/[unknown] frames mean you need frame pointers or DWARF unwinding, not a code fix.
Capture with perf record -F 99 -g; the workflow turns “it’s slow” into “it’s this function” in minutes — the proof step behind trace-and-prove.

mmap vs read(): When Mapping a File Actually Wins

Mon, 22 Jun 2026 00:00:00 GMT

“Just mmap it, it’s faster” is one of those pieces of advice that’s right often enough to be dangerous. Memory mapping can be a big win or a quiet loss, and the difference comes down to two costs you can reason about directly: syscalls and page faults.

What read() Actually Does

A read() copies bytes from the kernel page cache into your buffer:

char buf[64 * 1024];
ssize_t n;
while ((n = read(fd, buf, sizeof(buf))) > 0) {
    process(buf, n);
}

Every call is a syscall — a user→kernel transition — plus a memcpy from the page cache into your buffer. For a large file you pay that crossing many times, and you pay for the copy every time.

What mmap Does Instead

mmap maps the file’s pages into your address space. There’s no per-chunk syscall and no copy into a user buffer — you read the page cache directly:

struct stat st;
fstat(fd, &st);
char *data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);

for (size_t i = 0; i < st.st_size; i++)
    process_byte(data[i]);   // first touch of each page faults it in

munmap(data, st.st_size);

The cost moves from syscalls to page faults. The first access to each 4 KB page traps into the kernel, which maps the cached page into your tables. No copy, but a fault per page.

The Tradeoff in One Picture

   read():   [syscall][copy] [syscall][copy] [syscall][copy] ...
             └─ fixed cost per chunk, data copied into your buffer

   mmap():   [fault] . . . . [fault] . . . . [fault] . . . .
             └─ one fault per page, zero copy, paid lazily on touch

So the question is just: which is cheaper for your access pattern?

When mmap Wins

Random access into a large file. read/lseek per location is brutal; mapped memory makes it a pointer dereference.
Repeated passes over the same data. Pages fault in once and stay warm; no repeated copies.
Sharing the same file read-only across processes — they share the cached pages.

When read() Wins

Streaming once, front to back. You fault every page exactly once and never reuse it — that’s the same work as a copy, minus the prefetch friendliness of a plain sequential read. read() with a decent buffer (or readahead) is simpler and often faster.
Small files, where mmap/munmap setup and TLB churn outweigh any savings.
Pipes, sockets, and other non-seekable fds, which you can’t map at all.

Don’t Forget the Failure Modes

A read() returns -1 and an errno you can handle. A truncated mmap region hands you a SIGBUS when you touch a page past the file’s real end — an async failure that’s far easier to get wrong. If the file can change size under you, that risk is real.

Takeaways

read() trades syscalls + a copy; mmap trades page faults + no copy.
Map it for random access or repeated passes; stream it with read() for a single sequential pass or small files.
mmap can’t map pipes/sockets and turns I/O errors into SIGBUS — handle that before reaching for it.
Measure with your real access pattern; the winner flips depending on whether you touch each byte once or many times.

Anatomy of a Context Switch: What the Kernel Does Between Two Threads

Sat, 20 Jun 2026 00:00:00 GMT

Schedulers feel like magic until you profile a thread-heavy workload and watch real time disappear into __schedule. A context switch is not free, and the cost is not where most people think. Let’s trace exactly what happens between the instant one thread stops running and the next one starts.

What Counts as a Context Switch

Two very different things wear the same name:

Voluntary — a thread blocks (I/O, a lock, sleep) and hands the CPU back.
Involuntary — the scheduler preempts a thread that still wanted to run.

Both end up in the same kernel routine, but the trigger matters: involuntary switches cluster under contention and are the ones that wreck tail latency.

The Three Things the Kernel Saves

When schedule() decides thread B should run instead of thread A, it has to make A resumable later and B runnable now. Three pieces of state move:

// Simplified from kernel/sched/core.c
static void context_switch(struct rq *rq,
                           struct task_struct *prev,
                           struct task_struct *next)
{
    prepare_task_switch(rq, prev, next);
    switch_mm_irqs_off(prev->active_mm, next->mm, next); // address space
    switch_to(prev, next, prev);                         // registers + stack
    finish_task_switch(prev);
}

CPU registers — general-purpose registers, the stack pointer, and the instruction pointer get parked in thread A’s kernel stack.
The address space — if B belongs to a different process, switch_mm installs B’s page tables by writing CR3 on x86.
Bookkeeping — runqueue accounting, scheduler statistics, and FPU state (saved lazily on most CPUs).

Where the Cycles Actually Go

The register save/restore is cheap — tens of nanoseconds. The expensive part is the address-space change:

        thread A                 thread B
   ┌──────────────┐         ┌──────────────┐
   │ registers    │  fast   │ registers    │
   │ kernel stack │ ──────▶ │ kernel stack │
   └──────────────┘         └──────────────┘
          │  write CR3 (only if mm differs)
          ▼
   ┌─────────────────────────────────────────┐
   │  TLB entries for A may now be invalid    │  ← the real tax
   └─────────────────────────────────────────┘

Writing CR3 can flush the TLB. The next thread then takes a burst of TLB misses as it re-walks page tables for code and data it touches. That refill — not the register copy — is what you see in a flame graph.

This is why thread switches within the same process are cheaper than process switches: same mm, no CR3 write, TLB stays warm. It’s also why PCID (process-context identifiers) exists — tagging TLB entries so a switch doesn’t have to flush everything.

The Path, End to End

flowchart LR
    A[Thread A running] --> B{schedule called}
    B --> C[pick next: Thread B]
    C --> D[switch_mm: write CR3]
    D --> E[switch_to: swap registers + stack]
    E --> F[Thread B running]
    F --> G[TLB refills on first accesses]

Measuring It Yourself

perf makes the cost visible without guesswork:

# Count context switches for a command
perf stat -e context-switches,cpu-migrations ./your_workload

# See who is switching and why
perf sched record ./your_workload
perf sched latency --sort max

If context-switches scales with load while useful work stalls, you’re paying the scheduler tax — usually from lock contention or oversubscribed threads.

Takeaways

A context switch saves registers, may swap the address space, and updates scheduler bookkeeping — in that order of cost.
The register copy is cheap; the TLB refill after a CR3 write is the real expense. Same-process thread switches avoid it.
PCID/tagged TLBs exist specifically to soften that cost.
Reach for perf stat -e context-switches and perf sched latency before blaming the scheduler — most “scheduler overhead” is contention in disguise.

Floating Point: Why 0.1 + 0.2 Isn't 0.3

Mon, 15 Jun 2026 00:00:00 GMT

Type 0.1 + 0.2 into almost any language and you get 0.30000000000000004. This is the most reported non-bug in computing. The floating-point unit is working perfectly; the problem is that you asked it to store a number it physically cannot represent.

Computers Count in Binary Fractions

A float stores numbers as binary fractions — sums of powers of two:

   0.5    = 1/2           = 0.1   in binary
   0.75   = 1/2 + 1/4     = 0.11  in binary
   0.625  = 1/2 + 1/8     = 0.101 in binary

Any value that’s a sum of powers of two fits exactly. The trouble: 0.1 is not. In binary it’s a repeating fraction, like 1/3 is in decimal:

   0.1 (decimal) = 0.0001100110011001100110011… (binary, forever)

The hardware has finite bits (53 of significand in a 64-bit double), so it stores the closest representable value — slightly off. Add two slightly-off numbers and the errors surface in the last digits.

IEEE 754 in One Picture

A double splits its 64 bits like scientific notation in binary:

   ┌─┬───────────┬────────────────────────────────────┐
   │S│ exponent  │            significand             │
   │1│   11 bits │              52 bits               │
   └─┴───────────┴────────────────────────────────────┘
    sign  scale (×2^e)     the fractional digits

   value = (-1)^S × 1.significand × 2^(exponent − bias)

Roughly 15–17 significant decimal digits of precision. Plenty — until you assume it’s exact.

The Rules That Follow

Never test floats for equality. 0.1 + 0.2 == 0.3 is false. Compare within a tolerance instead:

if (fabs(a - b) < 1e-9)   // "close enough", not exact

(Real code should scale the tolerance to the magnitude of the values — an absolute epsilon is wrong for very large or very small numbers.)

Precision degrades with magnitude. The 53 bits are relative, so big numbers have coarse spacing. Past 2^53, integers themselves stop being representable — 9007199254740993.0 rounds to an even neighbor.

Don’t use floats for money. Repeated rounding accumulates. Use integer cents, or a decimal type that represents base-10 fractions exactly.

When You Need Exactness

Money / accounting → integers (cents) or a decimal library.
Comparisons → tolerance-based, scaled to magnitude.
Exact rationals → a big-rational type that stores numerator/denominator.

Floats are a deliberate trade: enormous range and speed in exchange for exactness. For physics and graphics that’s perfect; for a ledger it’s a liability.

Takeaways

Floats store binary fractions; decimals like 0.1 don’t fit, so they’re rounded to the nearest representable value.
IEEE 754 gives ~15–17 significant digits — relative precision, so large numbers are spaced coarsely (integers break past 2^53).
Never compare floats with ==; use a magnitude-scaled tolerance.
Use integers or a decimal type for money — never binary floats.

Bloom Filters: Probably Present, Definitely Useful

Fri, 12 Jun 2026 00:00:00 GMT

You want to know if a key has been seen before, but storing every key costs more memory than you have. A Bloom filter answers the membership question in a tiny fixed-size bit array — at the price of occasionally saying “yes” when the true answer is “no.” That one-sided error turns out to be exactly the trade real systems want.

A Bit Array and a Few Hashes

A Bloom filter is an array of bits, all zero to start, plus k independent hash functions. To add an element, hash it k ways and set those k bits:

   add("cat"):  h1→3  h2→7  h3→11
   bits: [0 0 0 1 0 0 0 1 0 0 0 1 0 0]
              3       7       11

To check membership, hash the same way and look at those bits:

   check("cat"): bits 3,7,11 all set?  → "probably present"
   check("dog"): any of its bits 0?    → "definitely absent"

If any required bit is 0, the element was never added — a guaranteed answer. If all are 1, it’s probably present.

The Asymmetry That Makes It Useful

This is the key property:

   "definitely not in the set"   ← always correct (no false negatives)
   "probably in the set"         ← may be wrong   (false positives possible)

A false positive happens when other elements collectively set all of a missing element’s bits. You never get a false negative — if it was added, its bits are set. So a Bloom filter never misses something it has seen; it only occasionally imagines something it hasn’t.

Tuning the Error

The false-positive rate is governed by the array size m, the number of elements n, and the hash count k. More bits per element → fewer false positives:

   ~10 bits per element, optimal k  →  ~1% false positives
   add more bits                    →  exponentially fewer

You pick the rate you can tolerate and size the array accordingly. Note: you can’t delete from a basic Bloom filter — clearing a bit might unset one shared with another element (a counting Bloom filter solves this).

Where They Earn Their Keep

The classic use is skipping expensive work. An LSM-tree database keeps a Bloom filter per on-disk SSTable. Before reading a file from disk to look for a key, it checks the filter:

   lookup(key):
     for each SSTable:
       filter says "absent"?  → skip the file entirely (no disk read)
       filter says "present"? → read it and check for real

A 1% false-positive rate means 99% of pointless disk reads are avoided for a few bits per key. The same pattern shows up in caches (“is this worth a lookup?”), CDNs, and spell-checkers.

Takeaways

A Bloom filter stores set membership in a bit array using k hash functions — far smaller than holding the elements themselves.
It never returns a false negative but can return a false positive: “definitely not” is exact, “probably yes” is not.
Size and hash count trade memory for a lower false-positive rate; basic versions can’t delete.
They shine as a cheap pre-check to avoid expensive work — like an LSM-tree skipping disk reads for absent keys.

Git Internals: Your Repo Is a Content-Addressed Database

Wed, 10 Jun 2026 00:00:00 GMT

Git’s command line is famously confusing. The data model underneath is not. It’s a small content-addressed key-value store with exactly four object types. Once you can see the objects, commands like reset, checkout, and rebase stop being incantations and become obvious manipulations of a graph.

Content Addressing: The Hash Is the Key

Git stores everything by the hash of its content. Hash the bytes, and that hash is the address:

echo 'hello' | git hash-object --stdin
# ce013625030ba8dba906f756967f9e9ca394464a

Same content always produces the same hash, so identical files are stored once. Change one byte and you get a completely different address — which is how Git detects any change and why history is tamper-evident.

The Four Objects

Everything in .git/objects is one of four types:

   blob    → file contents (no name, no metadata, just bytes)
   tree    → a directory: names + modes → blob/tree hashes
   commit  → a snapshot: one tree + parent(s) + author + message
   tag     → an annotated pointer to an object

They compose into the snapshot of your project:

   commit ──▶ tree ──▶ blob   (README.md)
                  └──▶ tree ──▶ blob   (src/main.c)
              parent
                │
                ▼
            commit ──▶ tree ──▶ ...

A commit doesn’t store a diff — it points at a tree, a full snapshot of every file at that moment. Unchanged files reuse the same blob hashes across commits, so snapshots are cheap despite being complete.

Commits Are a Linked List

Each commit records its parent(s), forming a directed graph back through history:

   A ◀── B ◀── C ◀── D   (main)
               ▲
               └── E ◀── F   (feature, branched at C)

A merge is just a commit with two parents. There’s no special “branch history” structure — branches are derived by walking parent pointers.

Refs: Branches Are Just Pointers

A branch isn’t a copy of anything. It’s a 40-character file containing the hash of one commit:

cat .git/refs/heads/main
# d4f7a... (the tip commit of main)

Creating a branch writes one tiny file. That’s why branching is instant. HEAD points at the current branch; committing just advances the branch’s pointer to the new commit. Now the commands fall out:

git reset moves a branch pointer to a different commit.
git checkout/switch moves HEAD and updates your working tree to that tree’s blobs.
git rebase replays commits onto a new parent, creating new commit objects.

Takeaways

Git is a content-addressed store: an object’s address is the hash of its content, so identical content is deduplicated and history is tamper-evident.
Four object types — blob, tree, commit, tag — compose into full snapshots, not diffs.
Commits link to parents to form the history graph; a merge is a two-parent commit.
Branches and HEAD are just pointers to commits, which is why branching and reset are instant.

eBPF: Running Your Code in the Kernel Without a Module

Thu, 04 Jun 2026 00:00:00 GMT

Traditionally, two options existed for kernel-level work: read what the kernel chose to expose, or write a kernel module and accept that a bug means a panic. eBPF adds a third: load your own program into the running kernel, attach it to an event, and trust that the kernel won’t let it crash anything. It’s the technology under bpftrace, Cilium, and most of the new observability stack.

A Virtual Machine Inside the Kernel

eBPF is a small register-based virtual machine built into Linux. You compile a restricted C program to eBPF bytecode, load it, and the kernel runs it in a sandbox when a chosen event fires:

   your .c  ──clang──▶  eBPF bytecode  ──load──▶  [ verifier ]
                                                       │ pass
                                                       ▼
                                     attached to an event, JIT-compiled,
                                     runs in kernel context

It is not a module. It can’t call arbitrary kernel functions, can’t loop unboundedly, and can’t touch memory it wasn’t handed.

The Verifier Is the Whole Trick

Before anything runs, the verifier statically analyzes every possible path through the program and rejects it unless it can prove safety:

   - all loops bounded (no infinite loops)
   - no out-of-bounds memory access
   - no uninitialized reads
   - bounded total instruction count

This is why eBPF is safe where modules aren’t: a program that might misbehave never loads. The cost is a constrained programming model — you write to please the verifier, which can be maddening, but it’s what makes running untrusted code in ring 0 acceptable.

Maps: Talking to User Space

An eBPF program is event-driven and short-lived, so it needs somewhere to keep state and a way to report results. Maps are kernel-resident key-value stores shared between the eBPF program and user space:

   kernel: eBPF program ──writes──▶ [ map ] ◀──reads── user-space tool
           (e.g. count syscalls)     (hash, array, ring buffer, ...)

Counters, histograms, per-PID stats — all live in maps. A user-space program polls them to draw the flame graph or latency histogram you actually see.

Where It Attaches

The power is the breadth of attach points:

   kprobes / tracepoints → trace any kernel function or event
   uprobes               → trace user-space functions too
   XDP                   → process packets at the NIC driver, pre-stack
   tc / cgroup hooks     → shape and filter traffic, enforce policy
   LSM hooks             → make security decisions

The same mechanism powers bpftrace one-liners, Cilium’s networking, and runtime security tools — one kernel feature, three industries.

# Count system calls by process, live, no kernel module:
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

Takeaways

eBPF runs sandboxed programs inside the kernel, attached to events — a safe alternative to kernel modules.
The verifier statically proves each program is bounded and memory-safe before it loads; that guarantee is the core of eBPF.
Maps are shared key-value stores that pass state and results between kernel and user space.
Broad attach points (kprobes, XDP, LSM) make one mechanism the basis of modern tracing, networking, and security tooling.

What a Container Actually Is: Namespaces and cgroups

Tue, 02 Jun 2026 00:00:00 GMT

People picture a container as a tiny virtual machine. It isn’t. There’s no guest OS, no hypervisor, no emulated hardware. A container is an ordinary process on the host kernel that has been lied to about what it can see and limited in what it can use. Two kernel mechanisms do all the work: namespaces and cgroups.

A Container Is Just a Process

Run a container and look on the host — you’ll find its processes in the normal process list, scheduled by the normal scheduler, sharing the host kernel:

   VM:                          Container:
   ┌─────────────┐              ┌─────────────┐
   │ guest kernel│              │   process   │
   ├─────────────┤              └──────┬──────┘
   │ hypervisor  │                     │ syscalls
   ├─────────────┤              ┌──────▼──────┐
   │ host kernel │              │ host kernel │  ← shared, no guest
   └─────────────┘              └─────────────┘

That shared kernel is why containers start in milliseconds and cost almost nothing — and also why isolation is weaker than a VM’s.

Namespaces: Lying About What Exists

A namespace gives a process its own private view of one kind of global resource. The kernel has several:

   pid     → its own process tree; PID 1 inside is some PID on the host
   mnt     → its own filesystem mounts (the "container's filesystem")
   net     → its own interfaces, routes, ports
   uts     → its own hostname
   ipc     → its own shared memory / semaphores
   user    → its own UID range; root inside ≠ root on host

Put a process in fresh namespaces and it sees a pristine system: PID 1, its own network stack, its own root filesystem. It believes it’s alone on the machine. Nothing is virtualized — the kernel just filters what the process can perceive.

cgroups: Capping What It Uses

Namespaces control visibility; control groups (cgroups) control consumption. They cap and account for CPU, memory, I/O, and process count:

   cgroup: web-app
     memory.max   = 512M     → OOM-killed if exceeded
     cpu.max      = 50%      → throttled past its share
     pids.max     = 100

This is what stops one container from eating the whole host. docker run -m 512m --cpus 0.5 is just writing these cgroup limits for you.

Docker Is the Convenience Layer

Strip it back and a container is: a process started in new namespaces, confined by cgroups, with its root filesystem swapped (pivot_root) to an unpacked image. You can build one by hand with unshare and cgcreate. Docker/containerd add the image format, layered filesystem, networking setup, and registry — enormous convenience, but not the isolation itself.

# A crude container by hand: new namespaces + a different root
sudo unshare --pid --net --mount --uts --fork --mount-proc \
    chroot /path/to/rootfs /bin/sh

Why It Matters

Because the kernel is shared, a kernel exploit from inside a container can reach the host — unlike a VM, where you’d also have to break the hypervisor. That’s the core security tradeoff, and the reason sandboxes like gVisor and Kata exist to put some of the VM boundary back.

Takeaways

A container is a host process, not a VM — same kernel, no guest OS, hence the millisecond startup.
Namespaces give it a private view of PIDs, mounts, network, etc.; it only thinks it’s alone.
cgroups cap its CPU, memory, and I/O so it can’t starve the host.
Shared kernel means weaker isolation than a VM — the tradeoff behind container security tooling.

Interrupts: How Hardware Taps the Kernel on the Shoulder

Tue, 26 May 2026 00:00:00 GMT

Polling is simple: ask the device “anything yet?” in a loop. It’s also how you turn a CPU into a space heater that checks an empty mailbox a billion times a second. Interrupts invert control — the device signals, the CPU responds — and that inversion is load-bearing for everything: your keyboard, your NVMe drive, your 100 Gbit NIC, and the scheduler tick that makes multitasking preemptive rather than cooperative. It also creates one of the kernel’s hardest constraint environments, which is why interrupt handling is split into halves.

The Mechanism

A device raises an interrupt (legacy IRQ line, or — for anything modern — an MSI-X message write, which is how one NVMe drive gets a separate vector per queue per CPU). The interrupt controller (APIC on x86) routes it to a CPU, which finishes the current instruction, saves state, and vectors through the IDT to the registered handler. Whatever was running — your process, another interrupt’s tail — is suspended mid-thought.

  NIC ──MSI-X──▶ APIC ──▶ CPU7: save state
                              └─▶ IDT[vector] ─▶ driver ISR   "top half"
                                     └─ raise NET_RX softirq  "bottom half, later"
                              ┌─▶ restore state
  interrupted task resumes ◀──┘

/proc/interrupts shows the whole routing table live — one row per vector, one column per CPU. A single column growing much faster than the rest is the classic “all my packets interrupt CPU0” smell; irqbalance or manual smp_affinity masks are the knob.

Why Handlers Are Split in Half

Interrupt context is a hostile place to run code. You can’t sleep (there’s no task to put to sleep — you’re borrowing someone’s stack), so no mutexes, no kmalloc(GFP_KERNEL), no page faults. And while you run, you delay everything — including other interrupts and any latency-sensitive task on that CPU. The rule is: do almost nothing.

So Linux splits the work. The top half (the ISR proper) does only what must happen at interrupt priority: acknowledge the device, grab the perishable data (read a status register, note which queue has work), schedule the rest, return. Microseconds. The bottom half does everything else — TCP processing, block completion — in softirq context, which runs with interrupts enabled, either on the tail of interrupt exit or, under sustained load, in the per-CPU ksoftirqd thread. That name is worth recognizing: ksoftirqd/3 eating CPU in top means “CPU 3 is drowning in deferred interrupt work” — usually network — and it’s the scheduler’s way of making that load visible and preemptible instead of invisible and latency-destroying. (Threaded IRQs and workqueues extend the same idea further for drivers that need to sleep.)

NAPI: Knowing When to Stop Listening

Interrupts have a failure mode at the high end: livelock. At a million packets per second, a per-packet interrupt means the CPU does nothing but enter and exit handlers — 100% busy, zero packets actually delivered to userspace. The fix, NAPI, is beautifully self-tuning: on the first packet, the driver’s top half disables RX interrupts and schedules a poll; the softirq then harvests packets in batches (budgeted, ~64 at a time) for as long as they keep coming; when a poll drains the queue, re-enable interrupts.

Light load: behaves like pure interrupts, minimum latency. Heavy load: behaves like pure polling, minimum overhead — one interrupt per burst, not per packet. The system slides between modes automatically, driven by the traffic itself. Hardware interrupt coalescing (ethtool -c) stacks on top of this, trading microseconds of latency for fewer vector deliveries; storage went the same direction, and io_uring’s polled mode drops interrupts entirely for latency-critical NVMe.

The last mile is placement: RSS spreads flows across NIC queues, each with its own MSI-X vector pinned to a CPU, so softirq work lands where the consuming application runs (RPS/RFS do it in software). Getting a 100 Gbit NIC to line rate is largely the craft of arranging which CPU gets tapped on the shoulder, and how often.

Takeaways

Interrupts preempt anything, anywhere; top halves therefore ack, stash, and defer — never sleep, never dawdle.
Bottom halves (softirqs) do the real protocol/completion work preemptibly; ksoftirqd burning CPU means deferred-interrupt overload.
NAPI flips interrupt→poll under load and back, defeating receive livelock with one elegant rule.
MSI-X gives modern devices per-queue vectors; /proc/interrupts + IRQ affinity decide which cores pay the tax.
Latency tuning is interrupt placement: coalescing, RSS/RPS, pinning — the shoulder-tap schedule is the performance model.

Journaling Filesystems: How ext4 Survives a Crash

Mon, 25 May 2026 00:00:00 GMT

A single file write isn’t one operation to the disk — it’s several: update the data blocks, the inode, the free-space bitmap, maybe a directory entry. Lose power between them and the filesystem is left in a contradictory state. A journal is how ext4 turns “several steps” back into something that’s all-or-nothing.

Why a Crash Corrupts

Appending to a file touches multiple structures that must stay consistent:

   1. write the new data blocks
   2. mark those blocks used in the bitmap
   3. update the inode's size and block pointers

Crash after step 2 but before 3, and the bitmap says “used” while no inode points there — leaked space. Crash between 1 and 2, and the inode may point at blocks the bitmap thinks are free, to be handed to another file later — corruption. The old fix was fsck scanning the entire disk on boot, which takes minutes to hours on a large volume.

Write-Ahead: Say It Before You Do It

A journal is a reserved on-disk area where the filesystem first records what it is about to do, then does it. The sequence:

   1. write the intended changes to the journal    ("transaction")
   2. mark the transaction committed                (a commit record)
   3. apply the changes to their real locations     ("checkpoint")
   4. mark the journal entry free

The commit record in step 2 is the atomic switch. After a crash, recovery just replays the journal:

   transaction committed?  ──yes──▶ replay it (redo the changes)
                           ──no───▶ discard it (it never happened)

Either the operation fully happened or it fully didn’t. Recovery reads a small journal, not the whole disk — seconds, not hours.

The Three Modes of ext4

Journaling everything is safe but doubles writes — data goes to the journal and its final home. ext4 lets you choose what’s protected:

   journal    metadata + data both journaled   safest, slowest
   ordered    metadata journaled; data written
              before the metadata commit         default, good balance
   writeback  metadata journaled; data unordered fastest, can show stale
                                                  data after a crash

Ordered mode (the default) is the clever compromise: it doesn’t journal file data, but it forces data blocks to disk before committing the metadata that points to them. So metadata never references garbage — you won’t see another file’s old contents — without paying to journal data twice.

The Limit

A journal guarantees the filesystem is consistent, not that your application’s data is. A committed transaction can still represent a half-written file from the app’s point of view. That’s what fsync() is for — and why databases call it religiously before reporting a write durable.

Takeaways

Filesystem operations touch several structures; a crash between them corrupts the volume.
A journal is a write-ahead log: record the intent, commit atomically, then apply — so recovery replays or discards whole transactions.
ext4’s ordered mode protects metadata and orders data before the commit, balancing safety and speed without journaling data twice.
Journaling protects filesystem consistency, not application durability — that’s still on fsync().

False Sharing: The Cache Line That Kills Your Parallelism

Sun, 17 May 2026 00:00:00 GMT

You parallelize a counter-per-thread loop. No locks, no shared variables — each thread writes its own slot. It should scale linearly. Instead, eight threads are slower than one. Welcome to false sharing: the performance bug where memory that isn’t shared behaves as if it is.

Cores Don’t Trade Bytes, They Trade Cache Lines

A CPU never moves a single byte between cores. The unit of coherence is the cache line — 64 bytes on most x86. When a core writes any byte in a line, the cache-coherence protocol (MESI) must invalidate that line in every other core’s cache:

   one 64-byte cache line
   ┌───────────────────────────────────────┐
   │ counter[0] counter[1] ... counter[7]  │   ← all on ONE line
   └───────────────────────────────────────┘
   thread 0 writes counter[0]  → invalidates the line everywhere
   thread 1 writes counter[1]  → invalidates it again

The variables are independent, but they ride the same line. Every write by any thread evicts the line from every other core, forcing a re-fetch across the cache hierarchy. The line ping-pongs between cores at memory speed.

Seeing It

The classic reproduction — an array of per-thread counters:

struct { long count; } counters[8];   // 8 longs = 64 bytes = ONE line

// thread i hammers counters[i].count
for (long j = 0; j < 100000000; j++)
    counters[i].count++;

This will scale negatively. The fix is to push each counter onto its own line:

struct {
    long count;
    char pad[64 - sizeof(long)];   // pad out to a full cache line
} counters[8];

Now each thread owns a private line — no invalidation traffic, and the loop scales the way you expected.

   before:  [c0 c1 c2 c3 c4 c5 c6 c7]          one contested line
   after:   [c0 ......][c1 ......][c2 ...] ... one line each

Doing It Cleanly

Hand-padding is error-prone. C++ gives you the line size and an alignment tool:

#include <new>
struct alignas(std::hardware_destructive_interference_size) Counter {
    long count;
};

In C, alignas(64) plus padding does the same job.

How to Catch It

You can’t see false sharing in the code — only in the hardware counters. perf exposes the coherence traffic:

perf c2c record ./your_program     # cache-to-cache transfer analysis
perf c2c report                    # points at the contended cache lines

A workload that should scale but flatlines, with high cache-coherence or HITM events, is the signature.

Takeaways

Cores exchange whole 64-byte cache lines, not individual variables.
False sharing happens when independent data lands on one line, so each write invalidates the line in every other core — turning parallel code serial.
Pad or align hot per-thread data to its own cache line to fix it.
Use perf c2c to find it; it’s invisible in source but obvious in coherence counters.

How malloc Works: Arenas, Bins, and Fragmentation

Sat, 09 May 2026 00:00:00 GMT

You call malloc(32) and get a pointer. That simplicity hides an allocator making a series of decisions: where the memory comes from, how free space is tracked, how it avoids locking on every call, and whether freeing it gives anything back to the operating system. Usually it doesn’t, and that surprises people.

The Allocator Sits Between You and the Kernel

The kernel deals in pages (4 KB) via brk (grow the heap) and mmap (map a fresh region). Asking it for every small object would be absurdly slow. So malloc grabs memory in big chunks and hands out small pieces itself:

   your code  ──malloc(32)──▶  allocator  ──brk/mmap──▶  kernel
                               (manages chunks, free lists)

The allocator’s whole job is to satisfy small requests from big slabs and reuse freed space without a syscall.

Bins: Free Lists by Size

When you free() a chunk, it isn’t returned to the OS — it goes onto a free list so the next malloc of a similar size can reuse it instantly. glibc groups these by size class into bins:

   fastbins:  [16][16]        small, recently freed, LIFO, no coalescing
   smallbins: [32][48][64]    exact size classes
   largebins: sorted ranges   for big allocations

A malloc(32) checks the matching bin first. Hit → O(1), no kernel involvement. This caching is why allocation is fast on average, and why a freed pointer used again (use-after-free) often still “works” — the memory’s still there, just re-handed-out.

Fragmentation: The Space You Can’t Use

Free memory that’s the wrong shape is useless. External fragmentation is scattered free chunks too small to satisfy a larger request even though the total is plenty:

   [used][free 16][used][free 16][used]   ← 32 bytes free total,
                                            but no contiguous 32-byte slot

Allocators fight this by coalescing adjacent free chunks back together and by carving from size classes that minimize waste, but it can’t be eliminated.

Why RSS Doesn’t Drop After free()

Memory freed back to the allocator generally stays in the process. Returning it to the OS requires the top of the heap to be free (to shrink brk), or the chunk to have come from mmap. A free list full of holes in the middle can’t be returned. So your resident set stays high even after freeing — it’s reserved for reuse, not leaked.

Arenas: Why Threads Don’t Serialize

If every thread shared one heap, every malloc would contend on one lock. glibc gives each thread its own arena (a separate heap with its own bins and lock), so concurrent allocations mostly don’t collide:

   thread 1 ─▶ arena A (lock A)
   thread 2 ─▶ arena B (lock B)     parallel, no contention

This is why allocator-heavy multithreaded code can still scale — and why replacing the allocator (jemalloc, tcmalloc, mimalloc) is a common performance win when the default’s arena strategy doesn’t fit your workload.

Takeaways

malloc buys memory from the kernel in big chunks and hands out small pieces from its own free lists.
Freed chunks go into size-classed bins for fast reuse — they’re not returned to the OS, which is why RSS stays high after free().
External fragmentation wastes space the wrong shape; coalescing helps but can’t fix it entirely.
Per-thread arenas avoid lock contention; swapping allocators is a real tuning lever.

The TLS 1.3 Handshake: What Those Milliseconds Buy

Sun, 03 May 2026 00:00:00 GMT

Every HTTPS connection begins with a negotiation that has to solve an absurd-sounding problem: two parties who’ve never met, over a wire an attacker fully controls, must agree on a secret key and verify who they’re talking to — in one round trip, because milliseconds are money. TLS 1.3 (RFC 8446, 2018) is the cleanest answer cryptography has shipped, mostly because of what it removed. Knowing its moving parts is the difference between debugging certificate errors by ritual and by reading.

One Round Trip, Three Jobs

  ClientHello ──▶   supported versions/ciphers, SNI,
                    + ECDHE key share (a guess, sent upfront)
              ◀── ServerHello (chosen cipher, server's key share)
                  ── everything below already encrypted ──
              ◀── EncryptedExtensions, Certificate,
                  CertificateVerify, Finished
  Finished    ──▶   application data flows

Job 1 — key agreement is ephemeral Diffie-Hellman (X25519 in practice): each side contributes a fresh keypair, both derive the same shared secret, an eavesdropper recording everything learns nothing. The 1.3 trick for speed: the client sends its key share speculatively in the first flight, gambling the server will accept X25519 (it will), collapsing 1.2’s two round trips into one. Note what’s gone: RSA key transport — 1.2’s mode where the client encrypted the secret to the server’s long-term key — which meant one leaked server key retroactively decrypted every recorded session. Ephemeral-only means forward secrecy is mandatory, and an entire class of both attacks and passive-decryption middleboxes died with it (enterprises now run explicit MITM proxies instead — at least the model is honest).

Job 2 — authentication rides the certificate chain: the server sends its cert plus intermediates; the client builds a path to a trusted root, checks names (SNI in, SAN out — CN has been dead for years), validity windows, and signatures; then — the step people forget exists — CertificateVerify: the server signs the handshake transcript with the certificate’s private key, proving it actually holds it. Without that, a certificate is just a public document anyone could replay.

Job 3 — integrity of the negotiation itself: both Finished messages MAC the entire transcript, so a middlebox stripping strong ciphers from the ClientHello (the downgrade attacks that plagued 1.2 — FREAK, Logjam) breaks the handshake. 1.3 also encrypts everything after the ServerHello, so certificates travel encrypted — network observers get SNI (until ECH lands widely) and little else.

Resumption, and the 0-RTT Footgun

Reconnecting clients get session tickets (PSKs) and skip certificate work — cheap and uncontroversial. The controversial gift is 0-RTT: sending application data in the first flight, encrypted under the resumed key, before any server response. Free latency — and structurally replayable: an attacker can capture the flight and deliver it again; no handshake-completion proof protects it. Safe only for idempotent requests, which is why serious deployments restrict it (CDNs allow GETs without bodies and reject the rest with 425 Too Early) and why “we enabled 0-RTT globally” belongs in a risk review, not a performance PR.

Where It Actually Breaks

Production TLS failures cluster in unglamorous places, none of them the cryptography: incomplete chains (server sends leaf without intermediate — works in browsers that cache/fetch intermediates, fails in curl/openssl/your language runtime; the source of “works in Chrome, fails in prod” forever; openssl s_client -connect host:443 -servername host shows exactly what’s sent), expiry (monitor -enddate programmatically; outages by calendar are self-inflicted), name mismatches (SNI-less clients hitting multi-cert servers get the default cert), and trust-store drift (containers FROM scratch, old CA bundles — ask the box, not the spec: curl -v names the failing step). The debugging kit is three commands deep: openssl s_client (handshake transcript + chain), openssl x509 -text (read the cert), and your load balancer’s config, because TLS terminates there and the backend hop is a separate security decision people forget to make.

Takeaways

1.3 = one round trip: speculative ECDHE share, chain + transcript signature for identity, transcript MAC against downgrade.
Forward secrecy is mandatory now — key compromise no longer decrypts history; passive-decrypt middleboxes are extinct by design.
CertificateVerify is why possession is proven, not just presented; everything after ServerHello travels encrypted.
0-RTT data is replayable by construction — idempotent requests only, enforced, or leave it off.
Real-world failures: chains, expiry, SNI/SAN, trust stores — all legible via openssl s_client before anyone blames the crypto.

Hash Tables: The Real Cost of a Collision

Fri, 01 May 2026 00:00:00 GMT

“Hash tables are O(1)” is true in the same way “the average human has one testicle” is true. The constant factor and the worst case are where programs actually live, and both are governed by how you handle collisions — two keys landing in the same slot.

Why Collisions Are Inevitable

A hash function maps a huge key space into a small array of buckets. By the pigeonhole principle, distinct keys will share buckets. With even a decent hash, the birthday paradox means collisions start appearing far earlier than intuition suggests — you don’t need the table to be full.

The fraction of slots used is the load factor (α = entries / buckets), and it’s the single number that governs collision rate.

Two Ways to Resolve Them

Chaining — each bucket holds a linked list of entries:

   [0] → (key1) → (key4)
   [1] →
   [2] → (key2) → (key7) → (key9)   ← collisions chain here

Simple and degrades gracefully, but each chain hop is a pointer chase to a random address — a likely cache miss. A “found in 3 comparisons” lookup can mean three cache misses, which dwarfs the comparisons.

Open addressing — entries live in the array itself; on collision you probe to the next slot:

   [0][k1][k4][ ][k2][ ][ ]   ← k4 wanted slot 1, probed to slot 2

Everything stays in contiguous memory, so probes are cache-friendly. The price: performance falls off a cliff as the table fills, because probe sequences get long. Open addressing needs a lower load factor (often kept under ~0.7).

The Cost You Forgot: Resizing

When the load factor crosses a threshold, the table doubles and every entry is rehashed into the new array — an O(n) operation. It’s rare, so amortized cost stays O(1), but any single insert can stall:

   insert, insert, insert, ... , INSERT(resize: rehash all n) , insert, ...
                                  └─ one slow op hidden in the average

For latency-sensitive code, that occasional spike matters. Preallocate with the expected size to dodge it.

What Actually Makes Them Fast

Keep the load factor sane. High α means long chains or long probes.
Mind cache behavior. Open addressing usually beats chaining in practice purely because it stays in contiguous memory.
Use a good hash. A weak hash clusters keys into the same buckets and turns your O(1) table into an O(n) linked list — the basis of algorithmic-complexity denial-of-service attacks.

Takeaways

Collisions are unavoidable; the load factor controls how often they happen.
Chaining degrades gracefully but pays cache misses; open addressing is cache-friendly but needs a lower load factor.
Resizing rehashes everything — amortized O(1) hides a real per-op spike, so preallocate when latency matters.
A bad hash collapses O(1) into O(n); pick a strong one.

HTTP/2 Multiplexing: One Connection, Many Streams

Thu, 23 Apr 2026 00:00:00 GMT

Under HTTP/1.1, a connection handles one request at a time. A slow response blocks everything behind it — head-of-line blocking — so browsers worked around it by opening six connections per origin and praying. HTTP/2 replaces the hack with real multiplexing. Then it runs into a wall TCP built.

HTTP/1.1’s Dilemma

One request, one response, in order, per connection:

   conn:  [── req A ──][ wait ][── req B ──][── req C ──]
                        └─ B and C stall behind a slow A

The six-connection workaround multiplies overhead — six TLS handshakes, six slow starts, six congestion windows — and the browser limit means request seven still waits.

Streams and Frames

HTTP/2 splits the single connection into many independent streams, each a request/response pair. Messages are chopped into frames tagged with a stream ID, interleaved on the wire, and reassembled on the other end:

   one TCP connection
   ┌──────────────────────────────────────────────┐
   │ [S1 hdr][S3 data][S1 data][S5 hdr][S3 data]…  │
   └──────────────────────────────────────────────┘
        frames from different streams, interleaved

Now a slow response on stream 1 doesn’t block stream 3 — their frames just interleave. One connection, one handshake, one warm congestion window, and effectively unlimited concurrent requests. The application-layer head-of-line problem is gone.

The Header Win

HTTP headers are repetitive — the same User-Agent, Accept, and cookies on every request. HTTP/2 adds HPACK, which keeps a shared dynamic table of previously sent headers so repeats become small references instead of full text. On a page firing 100 requests, that’s a real saving.

Where TCP Bites Back

Here’s the twist: HTTP/2 multiplexes streams, but they all ride one TCP connection, and TCP guarantees in-order byte delivery. If a single packet is lost, TCP holds back every stream’s bytes until the retransmission arrives — even streams that were complete:

   packet for S1 lost  →  TCP buffers everything after it
                          S3, S5 data sits ready but undelivered

The head-of-line blocking moved down a layer, from HTTP to TCP. On a clean network you never notice; on a lossy mobile link it hurts.

This is exactly the problem HTTP/3 solves by moving to QUIC over UDP, where each stream has independent delivery and one lost packet only stalls its own stream.

Takeaways

HTTP/2 carries many independent streams over one connection by interleaving frames, killing application-layer head-of-line blocking.
One connection means one handshake, one congestion window, and HPACK header compression — a big win over HTTP/1.1’s connection-juggling.
TCP’s in-order delivery reintroduces head-of-line blocking on packet loss, which is the motivation for HTTP/3 over QUIC.

Why Distributed Consensus Needs a Quorum

Wed, 15 Apr 2026 00:00:00 GMT

You replicate your data five ways for safety. A write lands on the leader — so which replicas must confirm it before you call it durable? Answer “one” and a crash loses data. Answer “all” and a single slow node freezes the system. The right answer is a majority, and the reason is a small, beautiful piece of arithmetic.

The Problem: Two Truths

The danger in a distributed system is split-brain — two halves of a cluster that lose contact and each decide they’re in charge. Both accept writes, and now you have two divergent versions of reality with no way to merge them.

To prevent this, every decision needs the agreement of a quorum: a subset of nodes large enough that two quorums can never both form at the same time.

The Majority Trick

Set the quorum to a strict majority — more than half. In a 5-node cluster, that’s 3. The key property:

   any two majorities of 5 must share at least one node

   {A B C}   and   {C D E}   overlap at C
   {A B C}   and   {A B D}   overlap at A, B

Because any two majorities overlap in at least one node, two conflicting decisions can’t both gather a quorum — the shared node would have to vote for both, and it won’t. Split-brain becomes mathematically impossible.

Read and Write Quorums

The same overlap logic lets you tune reads vs writes. With N replicas, a write quorum W and read quorum R, you get strong consistency when:

   W + R > N      →  every read quorum overlaps every write quorum
   W > N / 2      →  writes can't conflict with each other

For N = 5: W = 3, R = 3 is the balanced choice. Want faster writes? W = 2 breaks the guarantee. Want fast reads at the cost of slow writes? W = 5, R = 1 — but now any down node blocks all writes.

   N=5, W=3, R=3
   write → {A B C}
   read  → {C D E}     reads see C, which has the latest write ✓

Why Clusters Are Odd-Sized

A majority of 4 is 3 — the same as a majority of 5. So a 4-node cluster tolerates one failure, exactly like 3 nodes, while costing more and being likelier to have a failure. Odd sizes give you the best failure tolerance per node:

   nodes   majority   failures tolerated
     3        2              1
     4        3              1     ← extra node buys nothing
     5        3              2

This is why Raft and Paxos clusters are almost always 3 or 5.

Takeaways

A quorum is a subset big enough that any two quorums overlap — which is what makes conflicting decisions impossible.
A strict majority is the smallest such quorum; it prevents split-brain because two majorities always share a node.
W + R > N and W > N/2 give strong consistency; tune W/R to trade read vs write speed.
Use odd cluster sizes — an even size costs more without tolerating more failures.

Stack Canaries: How the Compiler Catches a Buffer Overflow

Tue, 07 Apr 2026 00:00:00 GMT

The oldest trick in memory-corruption exploitation is overflowing a local buffer to overwrite the saved return address, redirecting the CPU into attacker code. The stack canary is the defense that’s been quietly compiled into your binaries for two decades. It doesn’t prevent the overflow — it makes the program notice before it’s too late.

The Stack Layout Being Attacked

On a function call, the stack holds the local buffer, saved registers, and the return address the CPU will jump to when the function returns:

   higher addresses
   ┌────────────────────┐
   │  return address    │  ← overwrite this to hijack control
   │  saved frame ptr   │
   │  ...               │
   │  char buf[64]      │  ← overflow starts here, grows upward
   └────────────────────┘
   lower addresses

gets(buf) or an unchecked strcpy writes past buf, marches up through saved registers, and clobbers the return address. On ret, the CPU jumps wherever the attacker pointed.

The Canary

The compiler inserts a secret random value — the canary — between the local buffers and the return address. It’s written on entry and checked right before return:

   ┌────────────────────┐
   │  return address    │
   │  canary  (random)  │  ← any overflow reaching the return addr
   │  char buf[64]      │     must first overwrite this
   └────────────────────┘

A contiguous overflow that wants to reach the return address must trample the canary on the way. Before returning, the function compares the stored canary to the known-good value:

   __stack_chk_fail() if  stack_canary != fs:0x28

Mismatch → the program aborts immediately instead of returning into corrupted control flow. You’ve seen the result: *** stack smashing detected ***.

Why It’s Cheap and Where It Fails

The canary value lives in a per-thread location (the TLS, fs:0x28 on x86-64) and is randomized per process, so an attacker can’t just hardcode it. The runtime cost is a couple of instructions per protected function — the compiler only adds it to functions with stack buffers (-fstack-protector-strong).

It is not a cure. Attackers bypass canaries by:

Leaking the canary via a separate info-leak bug, then writing it back intact.
Overwriting non-contiguously — e.g. a bad array index that skips the canary and hits the return address directly.
Targeting other pointers (function pointers, GOT entries) that don’t sit behind the canary.

That’s why canaries ship alongside ASLR, NX/DEP, and CFI — defense in depth, not a single wall.

Takeaways

A stack canary is a random value placed between local buffers and the return address, checked before every protected return.
A linear overflow can’t reach the return address without destroying the canary first, so the program aborts instead of being hijacked.
It’s defeated by info leaks and non-linear writes — which is why it’s one layer among ASLR, NX, and CFI, not the whole defense.

Raft: Consensus You Can Actually Trace

Wed, 01 Apr 2026 00:00:00 GMT

I’ve written before about why quorums exist; this is the protocol layer above them. Raft (Ongaro & Ousterhout, 2014) was explicitly designed around understandability — the paper’s actual thesis — and it worked: etcd, Consul, CockroachDB, TiKV, and most Kafka deployments (KRaft) now bet their correctness on it. The payoff of learning it properly is that distributed-systems logs stop being mystical: term 847, leader election lost, proposal dropped are all lines you can trace back to specific rules.

The Frame: Terms and One Leader

Time divides into numbered terms; each term has at most one leader, elected by majority vote. All writes flow through the leader — Raft buys simplicity by refusing multi-master entirely. Every message carries the sender’s term; seeing a higher term instantly demotes you to follower. That one rule quietly handles most of the “stale leader” zoo: a partitioned ex-leader rejoins, hears term 848 > its 847, and steps down before doing damage.

Election: followers expect heartbeats; silence past a randomized timeout (150–300 ms class) triggers candidacy — increment term, vote for self, request votes. Majority → leader. The randomization is the whole liveness trick (simultaneous candidacies split votes; random timeouts make repeats improbable), and the vote-granting rule is the first safety pillar: a node refuses its vote to any candidate whose log is less up-to-date than its own. Elected leaders therefore already hold every committed entry — recovery requires no log transfer to the new leader, ever.

Replication: The Log Matching Property

The leader appends client commands to its log and ships them via AppendEntries, each carrying the previous entry’s (index, term) as a consistency probe:

  leader:   ... [5:t2] [6:t3] [7:t3]        AppendEntries(prev=6:t3, [7:t3])
  follower: ... [5:t2] [6:t3]           ✓ matches → append 7
  follower: ... [5:t2] [6:t2]           ✗ reject → leader backs up, overwrites

Induction does the rest: if a follower matches at (index, term), its entire prefix matches — divergent suffixes (from crashed leaders’ unreplicated tail entries) get found and overwritten mechanically. An entry is committed once replicated on a majority and — the subtle clause that fixed a real bug class (the paper’s Figure 8) — the leader only counts replication of entries from its own term; older-term entries commit implicitly behind them. Committed entries feed the state machine, and the guarantee stack — elected leaders have all committed entries + log matching + commit rule — yields the property everything else stands on: committed means durable across any minority of failures, in order, exactly once in the log (delivery to your application is another story — see below).

Where Deployments Actually Bleed

The protocol is the easy half; a decade of etcd/Consul incident reports names the hard one:

fsync is part of the protocol. Votes and log entries must hit stable storage before being answered — a node that votes, crashes, and forgets its vote can elect two leaders in one term. Slow disks therefore are slow consensus: etcd’s wal_fsync_duration_seconds alerting exists because commit latency is disk latency; running consensus on shared/burstable storage is a well-documented way to spend a weekend.
Elections during GC/CPU stalls. A leader that pauses past the timeout gets deposed; the cluster churns through terms (“leadership transferred” storms). Tuning timeouts against your worst-case pause — and using pre-vote (a candidate first checks it could win, without incrementing terms) plus leader stickiness — is what keeps a flaky node from term-bombing a healthy cluster. If your logs show terms climbing by hundreds, it’s this.
Membership changes are a protocol, not a config edit. Jumping straight from {A,B,C} to {A,B,C,D,E} lets two disjoint majorities exist mid-change. Raft’s answers (joint consensus, or one-node-at-a-time — what etcd enforces, now with learner/non-voting states to catch new nodes up safely) are why “just add two replicas at once” is refused by good implementations.
Reads need care too. Serving reads from any leader replica risks stale data from a deposed leader; linearizable reads use ReadIndex (confirm leadership with a quorum round) or leases — the reason etcd distinguishes --consistency=l from serializable reads, and a knob every Raft-backed store exposes somewhere.

Scope honesty completes the picture: Raft totally orders a log — it says nothing about exactly-once effects in your application (dedup/idempotency still yours), it costs a quorum round-trip per write (don’t put it on hot paths that don’t need it), and a majority loss halts writes by design — availability was traded for the guarantee, which is the correct trade for the configuration/metadata/ coordination tier it dominates.

Takeaways

Terms + majority elections + higher-term-wins handle stale leaders; randomized timeouts handle liveness.
Log matching lets one (index, term) probe verify entire prefixes; commit = majority replication of a current-term entry.
Consensus latency = fsync latency; monitor WAL sync times before blaming the network.
Pre-vote and single-node membership changes are what production Raft looks like; their absence is what incidents look like.
Raft orders the log — idempotency, read consistency modes, and the cost of quorum rounds remain your problems.

Harshith Sunku — Blog

An AI Coworker in an LXC Container

Why the agent gets its own container

The pipeline is the guardrail

What it’s actually like

Takeaways

Who's Allowed to Talk to What: Identity and Access in the Homelab

Identity gets its own machine — and its own segment

Tailscale is the front door

Design for the day identity is down

Takeaways

The Fleet: Six Machines, and What the Hypervisor Can't Do

The center, and one cheap trick worth stealing

The counters that wouldn’t count

A phone is the cheapest ARM dev board

The bench that has real ports

The machines that run the machines

Takeaways

Waking the Lab: an ESP32, MQTT, and 102 Bytes

The shape of the thing

MQTT choices that matter

The watchdog is the design

Takeaways

My Homelab: One Proxmox Box, ZFS Pools, and a Properly Segmented Network

The host, and the undo button

The VM/container line is a sizing lesson

The network does the isolation

Takeaways

How This Blog Runs for $0 a Month

The pipeline

Hosting: Cloudflare Pages

Search without a server

Social cards, fonts, CI — all build-time

The bill

What this architecture costs you instead

Takeaways

How LLM Inference Actually Runs: Tokens, KV Cache, and Why It's Memory-Bound

Generation Is a Loop, and Each Step Re-Reads the Model

The KV Cache: The Optimization Everything Depends On

Why Serving Looks the Way It Does

Takeaways

Reading Flame Graphs: Where Your CPU Time Actually Goes

Sampling: Statistics, Not Surveillance

Reading the Graph in Ten Seconds

The Traps That Mislead

Takeaways

mmap vs read(): When Mapping a File Actually Wins

What read() Actually Does

What mmap Does Instead

The Tradeoff in One Picture

When mmap Wins

When read() Wins

Don’t Forget the Failure Modes

Takeaways

Anatomy of a Context Switch: What the Kernel Does Between Two Threads

What Counts as a Context Switch

The Three Things the Kernel Saves

Where the Cycles Actually Go

The Path, End to End

Measuring It Yourself

Takeaways

Floating Point: Why 0.1 + 0.2 Isn't 0.3

Computers Count in Binary Fractions

IEEE 754 in One Picture

The Rules That Follow

When You Need Exactness

Takeaways

Bloom Filters: Probably Present, Definitely Useful

A Bit Array and a Few Hashes

The Asymmetry That Makes It Useful

Tuning the Error

Where They Earn Their Keep

Takeaways

Git Internals: Your Repo Is a Content-Addressed Database

Content Addressing: The Hash Is the Key

The Four Objects

Commits Are a Linked List

Refs: Branches Are Just Pointers

Takeaways

eBPF: Running Your Code in the Kernel Without a Module