/Programming

Everyone who touches a terminal has typed `ls | grep something` without giving it a second thought. And that's fine, most days it doesn't matter. But at some point it's worth opening that black box, because what's inside is a lot more interesting than "the pipe connects one program's output to another's input." When you call `pipe()` on Linux, the kernel doesn't create a file on disk or anything like that. It sets up a structure called `pipe_inode_info`, which manages a set of `pipe_buffer`s organized as a circular buffer. Each `pipe_buffer` points to a memory page — it's not one giant contiguous block like people sometimes picture. That already changes how you should think about what "writing to a pipe" actually means: you're not pouring bytes into a tunnel, you're filling pages that get referenced in a queue. The default size of that buffer is 65536 bytes, historically thought of as 16 pages of 4KB each. That number isn't arbitrary or set in stone — you can change it at runtime with `fcntl(fd, F_SETPIPE_SZ, new_size)`, and the reason the limit exists is simple: without it, a misbehaving process could hold onto kernel memory indefinitely just by stacking up data nobody reads. I've always found it kind of neat how a detail that sounds trivial is actually a design decision meant to prevent abuse of a shared resource. `pipe()` returns two file descriptors, one for reading and one for writing, but both point to the same `pipe_inode_info` underneath. And here's something I think gets underrated: pipes are treated by the Virtual File System as if they were files, even though they don't exist anywhere on disk. That fit into the VFS is what lets you use `read()` and `write()` on a pipe exactly like you would on a real file. Elegant, honestly. So what happens when the buffer fills up? The writing process gets placed on a wait queue and goes to sleep — literally taken off the run queue — until someone reads and frees up space. The kernel then calls something like `wake_up_interruptible` to wake whoever was waiting. Same thing in reverse: if the buffer is empty, a reader trying to pull data sleeps until there's something available. It's not polling, not a busy loop checking constantly; it's real synchronization happening inside the kernel, without your program noticing any of it. Worth a quick distinction between anonymous pipes and named pipes (FIFOs). The anonymous ones are what you use in the shell with `|`, they only exist while the processes are alive, and they never show up on the filesystem. FIFOs, created with `mkfifo`, get a path on the filesystem and can be opened by processes with no parent-child relationship at all — but under the hood they use basically the same `pipe_inode_info` structure. There's a part that tends to surprise even people who've used pipes for years: normally, moving data through a pipe means copying from kernel space to user space and back. But `splice()` exists, letting you move data between a pipe and another descriptor (another pipe, a socket, a file) without going through that intermediate copy. It's genuinely zero-copy. Tools like `nginx` and `tar` take advantage of this to squeeze out performance in heavy I/O scenarios, and it's one of the reasons understanding pipes under the hood isn't just academic curiosity — it has real weight in how you design a system. To close with something practical: when the shell parses `cmd1 | cmd2`, it calls `pipe()` to create the descriptor pair, then `fork()`s twice, and in each child process uses `dup2()` to redirect stdin or stdout to the right descriptor before calling `exec()`. Running `strace ls | grep something` and watching those syscalls happen live teaches more than any diagram would. And of course, there are the gotchas: if the reader dies before the writer, the writer gets `SIGPIPE` (or an `EPIPE` if it's ignoring the signal). Badly planned bidirectional pipes are a recipe for deadlock, since both sides can end up waiting on each other. And there's `pipe2()`, a variant that already accepts flags like `O_NONBLOCK` and `O_CLOEXEC` right at creation, skipping an extra `fcntl()` call later. In the end, the `|` we type without thinking is a stack of pretty specific design decisions — memory in pages, wait queues, zero-copy where possible. Worth peeking under the hood every once in a while.