[{"content":"","date":null,"permalink":"https://peripragmaton.com/tags/compilers/","section":"Tags","summary":"","title":"Compilers"},{"content":"","date":null,"permalink":"https://peripragmaton.com/tags/performance/","section":"Tags","summary":"","title":"Performance"},{"content":"Notes on performance: the places where the hardware, the compiler or the runtime stops behaving like the model most of us carry in our heads, and what that costs. Each post comes with the code and the benchmark that produced its numbers, in darfire/bloggy-code.\n","date":null,"permalink":"https://peripragmaton.com/","section":"PeriPragmaton","summary":"","title":"PeriPragmaton"},{"content":"An optimizing compiler is one of the more impressive pieces of software most of us use every day. Between parsing your code and emitting instructions, clang will run it through a hundred-odd passes: inlining, loop transformations, vectorization, instruction selection, register allocation, block placement. Each of these passes rests on decades of research. It\u0026rsquo;s easy to treat the whole thing as a black box that turns C into something fast. Most of the time that\u0026rsquo;s fair.\nThe compiler does have one blind spot, though, and it\u0026rsquo;s built into the job. It has to produce one binary that\u0026rsquo;s correct for every input, and for most decisions it has no idea which inputs will actually show up. This post is about what that costs and how to hand the compiler the missing information.\nWhat the compiler has to guess #Take a branch like this one:\nif (ins.flags \u0026amp; STAGE_3_RARE_CASE) { /* a dozen lines of fallback work */ } To lay this out in memory, the compiler has to put one side of the branch straight after the test and jump to the other. One side becomes the fall-through path and the other becomes the taken jump. If the fall-through side is the common one, the CPU fetches straight-line code. If not, every execution pays for a taken branch, and the hot code ends up spread across more cache lines than it needs, interleaved with code that rarely runs.\nWithout more information, the compiler estimates branch probabilities from static heuristics. Loop back-edges are probably taken. A comparison of a pointer against null probably fails. A path that ends in a call to abort() is probably cold. These rules work surprisingly well on typical code. They have nothing to say about ins.flags \u0026amp; 8, which looks 50/50 to the compiler. Whether that bit is set 2% of the time or 98% of the time depends on data the compiler never sees.\nThe usual manual escape hatch is __builtin_expect (or C++20\u0026rsquo;s [[likely]]). It works, but it\u0026rsquo;s a hint written by a person, at the time the code was written, about a workload that may since have changed. And it only covers the branches someone thought to annotate.\nWhy it matters this much is a front-end story, covered in more detail in an earlier post on instruction cache pressure. A modern core can execute far more than it can fetch through a fragmented instruction stream. Taken branches cost fetch bandwidth. Code that\u0026rsquo;s hot but spread across many lines pushes other hot code out of the 32KB L1 instruction cache and out of the op cache. None of this shows up in an instruction count.\nThe obvious fix is to stop guessing and measure. Profile-guided optimization (PGO) does exactly that: build the program once with instrumentation, run it on a workload that looks like real use, and feed the recorded branch and call counts back into a second compile. The compiler makes the same decisions as before, but with measured numbers instead of heuristics. The idea is old - GCC has supported it for more than two decades - and the tooling is mature.\nThe three steps #Taking clang as the target compiler, it\u0026rsquo;s three steps:\n# 1. build with counters on every branch and call clang -O2 -fprofile-generate -o app_instr app.c # 2. run it on something representative; writes default_*.profraw ./app_instr typical-input # 3. convert the raw counters, then rebuild using them llvm-profdata merge -o app.profdata default_*.profraw clang -O2 -fprofile-use=app.profdata -o app app.c The instrumented build is noticeably slower, because every branch and call updates a counter. The training run only has to see the program\u0026rsquo;s behaviour; it doesn\u0026rsquo;t need to be long. Profiles from several runs can be merged in step 3.\nGCC has the same feature under the same flag names, with a different profile format. Several large projects ship PGO builds: CPython\u0026rsquo;s --enable-optimizations trains on its own test suite, Firefox and Chrome are built with it, and GCC can build itself this way with make profiledbootstrap.\nWhat changes in the compiled code #A profile turns guessed branch probabilities into measured ones, and every pass that consults them makes different decisions. The main ones:\nblock placement. The hot path through each function becomes the fall-through path. Rarely executed blocks move to the end of the function, out of the way. inlining. Call sites that are executed often get a much larger inlining budget. Call sites that are never executed get none. register allocation. When something has to be spilled to the stack, spill and reload code lands on the cold paths. size versus speed. Functions that never ran in training are optimized for size, and hot and cold functions are grouped into separate sections, so the hot ones share fewer pages and cache lines. The interesting consequence is that the code doesn\u0026rsquo;t only get rearranged. It\u0026rsquo;s different code: larger where the program spends its time and smaller everywhere else. That will show up in SQLite below.\nA toy interpreter #To isolate the effect, I wrote a small bytecode interpreter (pgo-dispatch). It\u0026rsquo;s a switch over 128 generated opcodes. Each opcode does eight cheap steps, and after each step a bit in the instruction decides whether to run a larger rare-case fragment:\ncase 17: { uint64_t a = r[3], b = r[9]; a = (a ^ 0x9e3779b97f4a7c15ull) + (b \u0026gt;\u0026gt; 1) + ins.imm; if (ins.flags \u0026amp; 1) { /* six lines of rare-case arithmetic */ a += b; } /* ... seven more steps like the first ... */ r[5] = a ^ b; break; } The shape is borrowed from real interpreters, where the fast path of an opcode is interleaved with type checks, overflow checks and error handling. The rare-case bits are set 2% of the time, but nothing in the source says so.\nWithout a profile, clang treats each of those checks as a coin flip and keeps the rare fragment where the source put it, right after the test. The hot path then jumps over every one of them. With a profile, the fragments move to the end of the function and the hot path becomes straight-line code:\nTwo workloads run on the same interpreter. Workload A only uses the first half of the opcodes and workload B only the second half; think of numeric code and string code in a scripting language. Within each half, opcode frequency is skewed, so a few opcodes are hot and most are warm. The program is built from short phrases that repeat, as real bytecode does. The phrases matter: with a random opcode stream, every dispatch is a mispredicted indirect jump, and that cost drowns out everything else.\nI built it with plain -O2, then with PGO trained on A, on B, and on both with the profiles merged. Each binary ran both workloads. All variants are built without PIE, and every run produces the same checksum. These numbers come from an AMD Ryzen 7 7840HS (Zen 4) with clang 18, best of five runs:\nrun on trained on time taken branches icache misses speedup A - 0.3302s 651.4M 22.9M 1.00x A A 0.2382s 172.8M 5.0M 1.39x A B 0.2840s 646.4M 17.7M 1.16x A A+B 0.2472s 181.5M 10.3M 1.34x B - 0.3057s 643.4M 21.6M 1.00x B A 0.2907s 643.2M 18.4M 1.05x B B 0.2388s 174.0M 5.2M 1.28x B A+B 0.2483s 173.1M 11.8M 1.23x With a matching profile, the run gets 1.28–1.39x faster. Instructions retired go from 5.76G to 5.34G, a 7% drop that\u0026rsquo;s mostly the jumps that are no longer needed. That\u0026rsquo;s nowhere near enough to explain the speedup. The counters show where the rest comes from. Without a profile the interpreter takes about ten branches per opcode, one jump over each rare fragment plus the dispatch. With a matching profile it takes 2.6. Instruction cache misses fall by a similar factor of four, because the hot path of each opcode is now contiguous and the rare fragments no longer sit in the middle of it.\nWith the wrong profile, most of that disappears. The binary trained on B treats all of A\u0026rsquo;s opcodes as never executed, so none of the layout work lands where A needs it, and taken branches stay where they were. What\u0026rsquo;s left (1.16x on A, 1.05x on B) presumably comes from the cold half being compiled smaller, plus other second-order effects. I don\u0026rsquo;t have a clean explanation for why the leftover gain is asymmetric between the two directions.\nThe merged profile gets most of the benefit on both workloads. The icache column shows what it costs: twice as much code is treated as hot, and it all shares the same 32KB, so misses end up about double those of a matching profile.\nSQLite #A toy built to have this problem isn\u0026rsquo;t evidence that real code has it. For a real program I used SQLite and its own benchmark, speedtest1 (bolt-sqlite).\nThat\u0026rsquo;s a reasonable candidate. Most of SQLite\u0026rsquo;s time goes into sqlite3VdbeExec, a bytecode interpreter that is one large switch over its roughly 190 opcodes. Around it sit a B-tree, a pager and a query planner, all branchy, general-purpose code that has to handle every schema and every query.\nspeedtest1 comes with several test sets, and I used four of them. These are the kinds of workload different users of the same library might have:\nmain: the default mix of inserts, indexed and unindexed selects, updates, deletes, joins and sorting orm: wide tables with machine-generated, ORM-style queries cte: recursive common table expressions, such as a sudoku solver and a Mandelbrot set written in SQL fp: floating-point values, conversions and aggregates I trained PGO on main, on cte, and on both merged, at a quarter of the benchmark size. Every binary ran all four test sets against an in-memory database. All binaries produce the same result hash (--verify).\nMeasured on the same machine as the toy, best of seven runs, taken branches and icache misses from perf stat:\ntest set trained on time taken branches icache misses speedup main - 1.750s 2107.7M 50.2M 1.00x main main 1.549s 999.6M 25.8M 1.13x main cte 1.659s 1540.8M 32.9M 1.05x main main+cte 1.559s 1087.4M 29.5M 1.12x orm - 0.775s 1061.3M 53.1M 1.00x orm main 0.716s 696.9M 34.2M 1.08x orm cte 0.752s 827.0M 38.6M 1.03x orm main+cte 0.702s 709.0M 33.9M 1.10x cte - 2.741s 4569.0M 43.6M 1.00x cte main 2.481s 3229.6M 33.9M 1.10x cte cte 2.284s 2063.7M 29.4M 1.20x cte main+cte 2.290s 2171.7M 35.7M 1.20x fp - 1.673s 1879.9M 15.5M 1.00x fp main 1.599s 1450.8M 13.3M 1.05x fp cte 1.628s 1512.6M 8.2M 1.03x fp main+cte 1.584s 1417.9M 13.3M 1.06x The pattern from the toy holds, at a smaller scale. A matching profile gives 1.13x on main and 1.20x on cte. Training on the other test set gives 1.05x and 1.10x, so less than half the gain carries over in one direction and about half in the other. Taken branches tell the same story: the main profile halves them on main, while the cte profile removes only a quarter.\nNeither orm nor fp was part of any training run, which makes them the more honest test. Both gain from either profile, but the main profile does better on both. main is the broader of the two training workloads, and more of what it exercises is also what other workloads use.\nThe merged profile is as good as the best single profile on every row, within noise: 1.12x against 1.13x on main, and 1.20x on cte. It is also the best of the three on both test sets that weren\u0026rsquo;t trained on. Merging cost much less than in the toy, where the two workloads shared no opcodes at all. Here the two profiles overlap heavily. The icache column still shows the cost (35.7M misses against 29.4M on cte), but it no longer shows up in the time.\nThe gains are smaller than in the toy, even when the counters move by similar factors. That\u0026rsquo;s expected: the toy was built so that nearly all of its time goes into fetching code, and SQLite does a fair amount of real work per branch.\nA look inside #It\u0026rsquo;s worth looking at what the profile did to the interpreter function itself (clang 18.1.8 builds; exact sizes vary between compiler versions):\nbinary sqlite3VdbeExec size instructions whole .text base 49.3KB 11,229 1012KB pgo-main 79.7KB 18,322 978KB pgo-cte 57.0KB 13,261 882KB pgo-main+cte 109.5KB 24,981 1021KB The hottest function grows, by 60% with the main profile, while the program as a whole shrinks. Thirty-seven functions that the base build calls from the interpreter loop are inlined into it in the pgo-main build, among them sqlite3BtreeInsert, sqlite3PutVarint and btreeGetPage. In the other direction, code that never ran in training is compiled for size. With the cte profile, which exercises less of SQLite, .text shrinks by 13%.\nThis also explains why a profile from one workload can still help a different one. Both profiles agree on a large common core: opening cursors, decoding records, comparing values, moving through B-tree pages. What they disagree about is how much of the code at the edges is hot.\nChoosing what to train on #Every result above comes down to the training run, which is the actual engineering problem in using PGO. Some observations from doing this:\nTrain on what the program does in production, not on the benchmark. A narrow profile wins on the workload it was trained on and gives much less elsewhere. If you train on your benchmark, it will tell you the result you trained it to give. When in doubt, merge. A merged profile of several representative workloads gives up little or nothing on each of them, and in the SQLite runs it was the best choice for the workloads nobody trained on. Profiles go stale. A profile records counts against the code as it was. When the code changes, clang matches what it can and ignores the rest (-Wprofile-instr-out-of-date tells you when this happens). The training run belongs in the build, not in a file committed once and forgotten. The training run is part of the build. That means it needs to be deterministic enough to reproduce, and fast enough to run in CI. CPython\u0026rsquo;s test-suite training is a reasonable model. An alternative to the instrumented build is sampling-based PGO (AutoFDO in clang and GCC). Instead of a special build, you profile the normal binary in production with perf, using the CPU\u0026rsquo;s branch recording, and convert that into a profile for the next build. It\u0026rsquo;s less precise, but the profile comes from real traffic instead of a training script, which addresses the main weakness above.\nProfiling as part of the build #A profile describes the workload at one point in time, and workloads drift: users adopt new features, data grows, traffic shifts. A profile from last year can quietly optimize for a program that no longer runs. That\u0026rsquo;s the real argument for profiling periodically and proactively, not just once and not only when something is visibly slow - the same point the cache aliasing post ended on. With the training run in the build, every release gets a fresh profile, and a profile that changes noticeably from one release to the next is itself a signal that the workload has moved.\nConclusion #PGO doesn\u0026rsquo;t make the compiler smarter. It gives the compiler a fact it can\u0026rsquo;t derive from the source: which way the branches actually go. For code limited by the front end, such as interpreters, databases, compilers and other large, branchy programs, that is worth 10–30% for the cost of one extra build and one training run. The part that takes real thought is choosing a training run that represents the program\u0026rsquo;s actual use.\nThe same information can also be applied later, to a binary that has already been linked, without rebuilding anything. That\u0026rsquo;s what LLVM\u0026rsquo;s BOLT does, and it\u0026rsquo;s the subject of the next post.\nReferences # Clang - Profile Guided Optimization - the flags used here, and the sampling-based variant GCC - Instrumentation Options - -fprofile-generate and -fprofile-use in GCC SQLite - speedtest1 - the benchmark program darfire/bloggy-code - pgo-dispatch and bolt-sqlite - full code and bench.sh for the numbers in this post ","date":"October 25, 2026","permalink":"https://peripragmaton.com/posts/pgo/","section":"Posts","summary":"What the compiler has to guess about your program, and what changes when it doesn\u0026rsquo;t have to. Profile-guided optimization on a toy interpreter and on SQLite.","title":"Playing with Profile-Guided Optimization"},{"content":"","date":null,"permalink":"https://peripragmaton.com/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"","date":null,"permalink":"https://peripragmaton.com/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":null,"permalink":"https://peripragmaton.com/tags/caches/","section":"Tags","summary":"","title":"Caches"},{"content":"","date":null,"permalink":"https://peripragmaton.com/tags/cpu/","section":"Tags","summary":"","title":"CPU"},{"content":"This post is about the instruction cache, and what happens when a hot loop runs through more code than fits in it. It\u0026rsquo;s also about how a design decision that looks purely structural - whether a program processes one item at a time or a batch at a time - can decide how fast it runs, in a way no compiler will fix for you. The last part looks at high-performance networking, where this decision has shaped the architecture of real packet processors.\nThe instruction cache #Caches don\u0026rsquo;t only hold data. The front end of the core - branch prediction, fetch, decode - pulls instructions through its own L1 instruction cache, split from the data side: on a recent x86-64 core that\u0026rsquo;s 32KB, 8-way, 64-byte lines, the same set-indexed structure as L1d, just holding code. In front of it sits a cache of already-decoded micro-ops (the op cache on AMD, the uop cache / DSB on Intel), a few thousand entries, and behind it the unified L2, a dozen or so cycles away. A miss in L1i is worse than its latency suggests: the out-of-order back end can keep working through a data miss as long as it has independent instructions, but an instruction fetch miss means there are no new instructions to work on, so the whole pipeline drains while it waits.\nMost of the time this is invisible - hot code is small, the fetch prefetcher (steered by the branch predictor) runs ahead of sequential code, and what\u0026rsquo;s hot stays resident. It stops being invisible once a loop cycles through more code than the cache can hold: with a roughly LRU replacement policy, a cyclic working set even slightly larger than capacity evicts every line just before it\u0026rsquo;s needed again, so the miss rate jumps from near zero to nearly every line.\nA benchmark, and a wrong turn #To make that concrete: two ways to run the same N small functions over a batch of floats. Fused - for each item, call all N functions on it before moving to the next item. Staged - for each function, call it on every item before moving to the next function. Same math per item either way, same order of operations, just different loop nesting.\nFused. One item runs every instruction, start to finish, before the next one begins - the window slides the whole way through and has to start cold again for each item.\nStaged. One instruction\u0026rsquo;s worth of code stays hot while every item takes its turn through it - only the first item pays to fetch it.\nThe first version dispatched through an array of function pointers - ops[j](x) in a loop. Fused came out close to 3x slower than staged, which looked like confirmation of exactly the theory above: fused repeatedly cycles through all N functions\u0026rsquo; worth of code, more than fits in L1i, while staged only ever needs one function\u0026rsquo;s code hot at a time. Case closed, except it wasn\u0026rsquo;t.\nChecking the actual counters told a different story. L1-icache-load-misses barely differed between the two versions - a couple thousand either way. What differed by a factor of 25 was branch-misses: fused sat around a 20% misprediction rate on the indirect call, staged under 1%. The slowdown was real. It just wasn\u0026rsquo;t instruction cache pressure - it was the CPU\u0026rsquo;s indirect-branch predictor failing to learn a repeating sequence of 96 different call targets, a completely different mechanism that happens to produce a similarly-shaped number.\nIsolating the actual effect #To take the branch predictor out of the picture, I replaced the function pointers with direct calls. The benchmark now generates 96 functions and calls each one by name, in both loop orders, so every call target is known at compile time and no lookup through a table is needed. Each function also has a genuinely distinct body. Identical bodies get folded into one by the toolchain, which would have quietly undone the whole setup. A direct call doesn\u0026rsquo;t need branch target prediction the way an indirect one does, so that confound goes away on its own.\nWith that fixed, branch-misses came back nearly identical between the two versions - noise, not signal. Icache-load-misses did not:\nnitems mode time icache misses speedup 2000 fused 0.0059s 4377 1.00x 2000 staged 0.0018s 1861 3.28x 20000 fused 0.0567s 40774 1.00x 20000 staged 0.0187s 1688 3.03x 200000 fused 0.5809s 287836 1.00x 200000 staged 0.1864s 6449 3.12x (full code in icache-pipeline if you want the generator script and bench.sh that produced this)\nThe miss counts are the part worth sitting with, more than the times. Fused\u0026rsquo;s icache misses scale roughly linearly with item count, because every single item re-triggers the same fetch churn across all 96 functions. Staged\u0026rsquo;s barely move - it pays for each function\u0026rsquo;s code once per stage, not once per item, so growing the batch past that fixed cost is close to free.\nWhy the compiler won\u0026rsquo;t do this for you #Turning fused into staged is a known loop transformation - loop distribution - and both GCC and clang implement it. Neither applies it here, even at -O3, with link-time optimization and every op marked __attribute__((const)): GCC 12 and clang 18 both emit one loop with 96 calls in it. Their cost models think about vectorization and data locality; none of them estimates whether a loop body\u0026rsquo;s code fits in the instruction cache.\nSo staging is a design decision, and not a local one. It decides whether work flows through the program one item at a time or in batches, and that shapes everything around it: function signatures that take arrays instead of single values, buffers between stages, how errors and early exits are handled. It\u0026rsquo;s cheap to choose early and expensive to retrofit, which is why the systems that depend on it are built around it from the start.\nWhere this shows up outside a microbenchmark #Nobody writes a 96-function synthetic pipeline by accident, but \u0026ldquo;many small handlers, dispatched across a large batch, more combined code than fits in L1i\u0026rdquo; is a real production shape, and there\u0026rsquo;s a well-known system built directly around avoiding it: FD.io\u0026rsquo;s VPP (Vector Packet Processing, originally from Cisco, now a networking dataplane used widely enough to be the basis of multiple routers and vSwitches).\nVPP structures packet processing as a directed graph of small nodes - dpdk-input, ip4-lookup, and so on - and processes an entire vector of packets through one node before moving to the next, instead of running each packet through the whole graph individually. The project is explicit that this is an instruction-cache decision: each node is deliberately kept small enough to fit inside L1i, and because the first packet in a vector warms that node\u0026rsquo;s code up, every packet after it in the same batch runs against an already-hot cache. That\u0026rsquo;s \u0026ldquo;staged,\u0026rdquo; at production scale, built for the exact reason this post\u0026rsquo;s benchmark shows a 3x gap.\nFused vs staged, in a router. The kernel side is simplified: NAPI already polls packets from the driver in batches, and since 4.19 the early receive path can pass lists of packets along, but most of the stack still runs one packet at a time.\nSo, what #The honest moral here isn\u0026rsquo;t really about instruction caches. It\u0026rsquo;s about the first measurement I believed. A 3x gap with a plausible-sounding explanation is still just a plausible-sounding explanation until a second, independent counter confirms it\u0026rsquo;s actually the mechanism you think it is. Branch misprediction and icache pressure can produce numbers that look the same from a distance - a stopwatch in one hand and a theory in the other, and nothing stopping you from pairing them up wrong.\nReferences # FD.io VPP - Software Architecture - graph-node design and the instruction-cache rationale behind vector processing FD.io VPP - Scalar vs Vector packet processing - the scalar/fused vs vector/staged comparison directly darfire/bloggy-code - icache-pipeline - full benchmark code, generator script, and bench.sh for the numbers in this post ","date":"October 17, 2026","permalink":"https://peripragmaton.com/posts/icache-pressure/","section":"Posts","summary":"What happens when a hot loop runs through more code than the instruction cache can hold, and why batching work is a design decision the compiler won\u0026rsquo;t make for you.","title":"Under Pressure: Mind the Instruction Cache"},{"content":" All non-trivial abstractions, to some degree, are leaky.\n— Joel Spolsky, The Law of Leaky Abstractions (2002)\nThis is a post about small, easy-to-miss changes with consequences nowhere near proportional to their size - the kind a code reviewer would wave straight through without a second thought. One of them, in particular, is going to cost eight times the runtime before this post is done, and no amount of reading the code will tell you why.\nThe gap between the language and the machine #Programming languages, more or less, still model something close to an idealized Turing machine - a tape, a pointer into it, and the assumption that moving the pointer and reading a cell costs the same no matter where on the tape you are.\nThat assumption is false on every CPU made in the last few decades, and it\u0026rsquo;s false in a bunch of different, unrelated ways at once:\ncaches make some addresses dramatically cheaper to touch than others depending on what you touched recently the TLB does the same thing for virtual-to-physical address translation instruction fetch has its own separate cache with its own separate failure modes hyperthreading means the \u0026ldquo;CPU\u0026rdquo; executing your instructions might not even have exclusive use of its own execution units while it\u0026rsquo;s doing it. And the list goes on. None of this is exotic or edge-case - it\u0026rsquo;s just what the hardware actually is, underneath the tape-and-pointer story the language tells you. If you care about runtime performance, rather than the theoretical one, you eventually have to know at least a little about this layer. Nothing above it is going to warn you when you\u0026rsquo;ve stepped on it.\nNext I\u0026rsquo;ll focus on showing practical effects of an obscure but important quirk of CPU caches: cache aliasing.\nA quick primer on cache hierarchy, and where aliasing comes from #The part of this that the rest of the post is actually about is caches specifically, so a quick refresher.\nA CPU doesn\u0026rsquo;t talk to RAM directly for every access - there\u0026rsquo;s a small hierarchy of progressively bigger, slower caches sitting between the core and main memory. Whichever one actually holds the data you want determines whether a read costs a handful of cycles or a couple hundred.\nRather than make up round numbers, here\u0026rsquo;s the actual hierarchy on the machine this post\u0026rsquo;s numbers come from. This runs on an AMD Ryzen 7 7840HS - Zen 4, codename \u0026ldquo;Phoenix,\u0026rdquo; a laptop/mobile part, not a server chip. Sizes are read straight from /sys/devices/system/cpu/cpu0/cache/; latencies aren\u0026rsquo;t something you can read out of sysfs, so those are published figures for this specific chip, converted to cycles at this machine\u0026rsquo;s ~3.79GHz:\nCache hierarchy: AMD Ryzen 7 7840HS (Zen 4, \u0026ldquo;Phoenix\u0026rdquo;). This post\u0026rsquo;s test machine. Bar width is illustrative, not to scale.\nL1d/L2 latency is a general Zen 4 core figure; L3/RAM latency is chip-specific, converted to cycles from measured nanoseconds at this machine\u0026rsquo;s own clock speed - sources are in the references section at the end.\nEach of those levels is carved up into fixed-size lines (64 bytes is typical), and the lines are grouped into sets. An address doesn\u0026rsquo;t get to live just anywhere in the cache - a slice of its own bits determines which set it\u0026rsquo;s allowed to land in, and a cache is only associative enough to hold so many lines per set at once (8-way is common for L1) before something has to get evicted to make room. Normally this works out fine, because normal access patterns spread themselves across plenty of different sets.\nCache aliasing is what happens when they don\u0026rsquo;t. If the stride between the addresses you\u0026rsquo;re touching happens to be a multiple of the cache\u0026rsquo;s entire set-indexing range, every single address you touch maps to the exact same set, no matter how far apart they are in memory. The cache can hold maybe 8 of those before it starts thrashing, and you\u0026rsquo;re asking it to hold however many you\u0026rsquo;re actually touching.\nThe classic way to back into this by accident is a power-of-two stride, which is exactly what a power-of-two row length gives you in a 2D array. These practical examples may look a bit contrived. But the effect is strongest exactly where we reach for power-of-two values for dimensions and strides - which is the default we\u0026rsquo;ve all been taught to use, for better or worse.\nReproducing it on a real machine #Take an N x N matrix of doubles, stored row-major, and sum it column by column instead of row by row:\nfor (j = 0; j \u0026lt; n; j++) for (i = 0; i \u0026lt; n; i++) sum += a[i * ld + j]; ld here is the row length - the leading dimension, in the usual terminology. With no padding, ld equals n, and if n is a power of two, every step of that inner loop lands in the same set:\n8x8 matrix, row length 8 (pow2 stride). set index = column number\nScaled down to a toy 8-set cache instead of a real 64-set one so it fits on screen, but the arithmetic is identical. Row length 8, and the set index is just the column number - every row repeats the same 8 colors in the same order, so the whole column piles into one set, over and over, for every single row.\nThe fix is to pad the row by one unused element. ld becomes n + 1, the stride stops being a round power of two, and the exact same 8 colors spread out instead:\n8x8 matrix, row length 9 (padded). set index = (row + col) mod 8\nOne extra double per row, allocated and never read, and the set index becomes (row + col) mod 8 instead of just col - one of each color per row and per column now, instead of a whole column piling into one.\nTo actually measure this rather than just reason about it, I used perf stat with cache-references and cache-misses alongside wall-clock timing. That\u0026rsquo;s all wrapped in a small script (bench.sh in the repo): it builds both variants, runs each one a handful of times per size to smooth out noise, and reports the miss rate and the speedup relative to the slow case. Numbers from actually running it (full code in matrix-cache-aliasing if you want to poke at it):\nN access pattern LD time miss % speedup 512 row-major (baseline) 512 0.0043s 0.95% 6.37x 512 column, pow2 LD 512 0.0274s 40.44% 1.00x 512 column, padded LD 513 0.0031s 0.44% 8.84x 1024 row-major (baseline) 1024 0.0026s 0.53% 9.04x 1024 column, pow2 LD 1024 0.0235s 37.75% 1.00x 1024 column, padded LD 1025 0.0030s 0.19% 7.83x 2048 row-major (baseline) 2048 0.0030s 0.47% 8.33x 2048 column, pow2 LD 2048 0.0250s 25.29% 1.00x 2048 column, padded LD 2049 0.0092s 3.99% 2.72x 4096 row-major (baseline) 4096 0.0120s 0.19% 22.20x 4096 column, pow2 LD 4096 0.2664s 21.59% 1.00x 4096 column, padded LD 4097 0.0441s 5.88% 6.04x Same data, and arguably the same order of operations too, since padding doesn\u0026rsquo;t change what gets summed, only where each element happens to sit in memory. One version still ends up an order of magnitude slower, for reasons you would not find by just reading the loop.\nA more realistic example: matrix multiply #The column sum makes the point cleanly, but it\u0026rsquo;s also obviously a demonstration - nobody accidentally writes a column-sum microbenchmark. A naive matrix multiply is closer to something most people write at some point without thinking twice about it, often as a first pass before reaching for a real linear algebra library:\nfor (i = 0; i \u0026lt; n; i++) for (j = 0; j \u0026lt; n; j++) for (k = 0; k \u0026lt; n; k++) c[i][j] += a[i][k] * b[k][j]; b[k][j] is the same column-stride access as before, just one level deeper. Now it\u0026rsquo;s sitting inside an O(n^3) loop instead of an O(n^2) one, so whatever it costs you gets multiplied out accordingly (matmul-cache-aliasing):\nN layout LD time miss % speedup 256 pow2 LD 256 0.0275s 1.18% 1.00x 256 padded LD 257 0.0094s 0.30% 2.93x 512 pow2 LD 512 0.7277s 60.82% 1.00x 512 padded LD 513 0.0841s 5.07% 8.65x 1024 pow2 LD 1024 6.3079s 49.36% 1.00x 1024 padded LD 1025 0.8995s 1.90% 7.01x Up to 8.65x faster at N=512, from allocating one column you never actually use for anything.\nMaybe this is only a hand-rolled-C problem #Both examples so far are C, which raises an obvious question: is this a \u0026ldquo;writing your own loops\u0026rdquo; problem, or does it show up in the libraries people actually reach for day to day?\nI tried reproducing it in numpy to find out, and the first two things I tried didn\u0026rsquo;t work at all. a.sum(axis=0) comes back basically the same speed either way. That\u0026rsquo;s because numpy\u0026rsquo;s reduction iterates the outer axis and vectorizes over whichever rows are contiguous, regardless of which axis you told it to reduce along - it never actually walks down a column one element at a time, no matter how you ask for it. a @ b barely moves either, even with the plain reference BLAS this machine happens to have installed rather than OpenBLAS or MKL, because matmul implementations are specifically built around not doing this. Which is at least a partial answer to the question: production numerical libraries do guard against this, deliberately.\nWhat does reproduce it is a.T.copy(). Transposing a view and forcing it contiguous means numpy has to fill the result row by row - which means reading the original column by column, exactly the access pattern from the first example:\nN layout time speedup 1024 pow2 0.0062s 1.00x 1024 padded 0.0008s 7.50x 2048 pow2 0.0440s 1.00x 2048 padded 0.0104s 4.24x 4096 pow2 0.5080s 1.00x 4096 padded 0.0432s 11.76x Noisier than the C version - the speedup doesn\u0026rsquo;t scale smoothly with N the way the C benchmarks mostly do, it jumps around. But the effect is real, and in the same direction, at every size tested. So: numpy\u0026rsquo;s own built-in reductions and its matmul are both guarded, deliberately, by people who already knew about this. A plain array operation that happens to walk memory the wrong way is not.\nSo, proactively, then #None of this throws an error. The column sum still adds up to the right total, the matmul still produces the right matrix, a.T.copy() still gives you a correctly transposed array - every one of these examples is functionally correct the whole time, slow or fast. That\u0026rsquo;s most of why it\u0026rsquo;s easy to miss: there\u0026rsquo;s no exception, no warning, nothing that distinguishes the pathological version from the fine one except a stopwatch.\nThe examples in this post are deliberately extreme, chosen to make the effect impossible to miss. But the underlying mechanism doesn\u0026rsquo;t need an 8x or 9x gap to be real. A buffer that\u0026rsquo;s merely inconvenient-sized can lose you a smaller, quieter percentage that never gets big enough to trip the \u0026ldquo;something is obviously wrong\u0026rdquo; threshold at all - and just sits there as a permanent, invisible tax.\nProfiling, in most places, is something you reach for reactively - after a thing is already visibly, unambiguously too slow, usually once you\u0026rsquo;ve ruled out the easier explanations. Compare that to how we treat correctness. Nobody waits for a bug report before running the test suite; tests run on every commit, whether or not anything looks broken, precisely because \u0026ldquo;looks fine\u0026rdquo; and \u0026ldquo;is fine\u0026rdquo; aren\u0026rsquo;t the same claim. Performance gets no equivalent default. Most code ships having never been profiled at all, not because it\u0026rsquo;s fast, but because nothing about it looked slow enough to bother checking.\nThat asymmetry is the actual point of this post, more than any specific number in any specific table. Profiling only reactively means only catching the bugs that are already expensive enough to notice - and cache aliasing, almost by design, doesn\u0026rsquo;t announce itself that way until it\u0026rsquo;s too late to be a quick fix.\nThe answer isn\u0026rsquo;t running perf on everything, which isn\u0026rsquo;t realistic and isn\u0026rsquo;t really the point. It\u0026rsquo;s treating a baseline profiling pass - at least once, on something closer to the real deployment hardware than a laptop - as a normal, disciplined part of shipping code that touches memory at any real scale, the same unglamorous way a test suite is just a normal part of shipping code that\u0026rsquo;s supposed to be correct.\nReferences # Ulrich Drepper - Memory part 2: CPU caches - from his \u0026ldquo;What Every Programmer Should Know About Memory\u0026rdquo; series; the canonical deep dive on cache associativity, conflict misses, and exactly this kind of power-of-two-stride aliasing TechPowerUp - AMD \u0026ldquo;Zen 4\u0026rdquo; Dies, Transistor-Counts, Cache Sizes and Latencies Detailed - L1d/L2 latency figures Chips and Cheese - Hot Chips 2023: AMD\u0026rsquo;s Phoenix SoC - L3/DRAM latency figures for this machine\u0026rsquo;s actual chip (Ryzen 7 7840HS, \u0026ldquo;Phoenix\u0026rdquo;) darfire/bloggy-code - matrix-cache-aliasing and matmul-cache-aliasing - full benchmark code and bench.sh for the numbers in this post ","date":"October 11, 2026","permalink":"https://peripragmaton.com/posts/cache-aliasing/","section":"Posts","summary":"Summing a power-of-two-sized matrix column by column is eight times slower than the same loop with one extra element per row. Why cache sets make some addresses collide, and how to spot it.","title":"Quirks of a CPU: Cache Aliasing"},{"content":"Peri pragmaton - περὶ πραγμάτων - is Ancient Greek for \u0026ldquo;about things\u0026rdquo;, or \u0026ldquo;concerning matters\u0026rdquo;.\nΤαράσσει τοὺς ἀνθρώπους οὐ τὰ πράγματα, ἀλλὰ τὰ περὶ τῶν πραγμάτων δόγματα.\nTarassei tous anthrōpous ou ta pragmata, alla ta peri tōn pragmatōn dogmata.\nMen are disturbed not by things, but by their opinions about things.\n— Epictetus, Enchiridion 5\n","date":null,"permalink":"https://peripragmaton.com/about/","section":"PeriPragmaton","summary":"","title":"About the name"}]