The Upstream and Downstream of AI: When 'Saving Memory' Is a Headline You Have to Take Apart

A long data center cold aisle in deep perspective, front racks brightly lit and the far end fading into darkness, an engineer crouched in the foreground routing a bundle of fiber

Thirty spokes share one hub; it is the empty space at the center that makes the cart useful. Clay is shaped into a vessel; it is the emptiness within that makes the vessel useful. Doors and windows are cut for a room; it is the emptiness that makes the room useful. Therefore what exists gives advantage, but what does not exist gives use.

— Tao Te Ching, Chapter 11

Laozi is making a counterintuitive point. A wheel turns not because of its thirty spokes but because of the hole at its center. A house shelters you not because of its walls but because of the space those walls enclose. The solid parts provide the advantage; what actually gets used is the part where nothing is.

That maps unusually well onto today’s compute industry. When people talk about AI, they talk about the computing — how many chips, how many operations, how large the model. But this episode is about the other thing entirely: the empty parts. Memory, bandwidth, interconnect, which tier you park your data in. Those are what determine whether the cart turns at all. And the most valuable thing in this episode is what happened when someone tried to make that emptiness smaller: reality did not follow intuition.

What This Episode Covers

The hosts put three seemingly unrelated events from the previous week side by side.

First, Moonshot’s new open-weight model, Kimi K3, whose headline feature is a large reduction in key-value cache footprint — the intermediate state a model holds while generating — plus a new attention algorithm. Second, the World AI Conference held the same week, where Chinese vendors showed a thousand-card-class superpod and a domestically developed chip claiming advanced-packaging-grade capability. Third, US earnings season: a server assembler’s gross margin guidance revised sharply upward, Google raising capital expenditure again while booking external revenue from its own silicon for the first time, and Texas Instruments discussing progress on 800-volt power delivery.

The hosts didn’t treat these as three separate news items. They had one question: along the chain from model to chip to memory to power delivery, which segment is tightest right now, and where is the money actually flowing?

A small aside from the opening, which I enjoyed: the hosts spent a while puzzling over the relationship between “Kimi” and the company’s Chinese name, and concluded there simply isn’t one. It’s a Chinese company, the Chinese name came first, and the English name was chosen separately.

Key Points

1. The bill for a cheap model only makes sense after you multiply three numbers together.

The episode mentions someone comparing this open-weight model against a top-tier flagship on the same task. The result: unit price is roughly a third, but because its raw capability isn’t at flagship level, it takes detours on mid-sized tasks — first attempt fails, tries again — so total tokens burned come out around double. On top of that, Chinese models have historically been slower at generating tokens, so wall-clock time stretches too. Multiply all three and the total is still cheaper.

The point isn’t “it’s cheaper.” The point is that those three numbers define what it’s for. Interactive chat is out, because people won’t wait. But anything that runs in the background and doesn’t need an immediate reply — nightly report compilation, scheduled jobs, repetitive agent-style work — doesn’t care whether it finishes in ten seconds or half an hour. That’s its home turf. This is also why several US companies already list Chinese models among their defaults: not because they think they’re better, but because they’ve worked out which jobs don’t need to be paid at flagship rates.

2. Be precise about which block the 75% applies to.

The 75% reduction is in key-value cache, not total memory. The mechanism is called Kimi Delta Attention — one host joked the name sounds cool because it has a Delta in it, and it turns out the Delta genuinely means the mathematical difference: rather than storing the full state repeatedly, it stores the change and reconstructs from it. It’s mixed with conventional full attention layers at a three-to-one ratio, which is exactly where the 75% figure comes from. The arithmetic is that straightforward.

But inference itself also consumes memory, and that’s a separate pool from the cache. As noted above, this model burns roughly double the tokens to complete an equivalent task — and that doubled inference pass consumes memory all the same. So “it saves cache, therefore we need less memory” mistakes a saving in one block for a saving overall. The episode mentions people genuinely assuming they could run it on a consumer graphics card; the host’s response was blunt — maybe it stumbles along at zero context. The official recommendation is sixty-four accelerator cards.

There’s one more point that’s easy to skip past but carries real information: this model was trained using an open low-precision format rather than a specific vendor’s proprietary one. That means anyone fine-tuning it or building a next generation on top of it isn’t necessarily locked to one silicon vendor. That’s optionality, not performance — invisible in the short term, structurally relevant to bargaining power over time.

3. Open weights don’t mean you can run it.

What does sixty-four accelerator cards actually mean? Roughly eight full racks chained together. One host looked up the price mid-conversation and revised himself, landing at about two to three million US dollars per rack — while noting that might be a factory price excluding the rest of what you’d need, so real deployment costs more. His conclusion: this is still enterprise territory, still some distance from genuine on-premises self-hosting. It lowers the threshold a bit; it doesn’t remove it.

Worse, mixture-of-experts architectures are extremely sensitive to interconnect speed. Every token has to move all-to-all across cards, so if you spread eight racks apart and link them with slower cabling to save money, the cost doesn’t disappear — it just relocates from silicon to networking. Open weights solve licensing, not physics.

4. Anyone can name a spec. Look at the object.

When the episode turns to a Chinese vendor announcing an advanced-packaging-grade chip and a glass substrate, the host’s reaction is telling: you can call it whatever you like, names are free — what I want to see is the cross-section and the spec band. In the same stretch they unpack what’s actually hard about a superpod. Connecting racks isn’t hard. Making 1,024 cards talk to each other without losing speed is. Switching, topology, the split between network interface and switching silicon — that’s the real barrier. When one company scales to four thousand-plus units with no speed degradation, that’s the impressive part, not the number itself.

There’s also a clear observation about the Chinese approach: constrained on process, they compensate — slower per lane is fine, so add more pins, double the I/O. Someone tore down a chip and reported it was built with multiple exposure passes: yield is poor, but it works. Trading money for yield, trading area for speed. That’s an explicit tradeoff, not black magic.

5. Storage in place of compute: memory is no longer one thing, it’s a stack.

This is the most technically dense and most worth remembering part of the episode. Since the cache is what eats memory, people started offloading it downward: tier one is high-bandwidth memory, tier two is low-power memory on the board, tier three is local or remote conventional memory, and below that, solid-state storage. The principle is one sentence — the less frequently you need it, the further away it goes.

Chinese vendors are especially fond of this “storage in place of compute” framing, for a very practical reason: their high-bandwidth memory supply is still catching up, so they have to find substitutes lower down the stack. But the same direction is being pursued in the US with a different motive. High-bandwidth memory and conventional DRAM compete for the same process capacity; flash sits on a separate line. Using more of the latter routes around the most congested path.

One nicely human moment here: the episode mentions a Japanese reporter catching a memory maker’s president and asking what he’d just discussed with a chip company’s chief executive, and whether memory came up. The president laughed. Asked about the offloading roadmap specifically, he laughed harder and said he couldn’t comment. The comment section’s conclusion: progress is clearly going fine.

On volume, the figure cited is flash demand of around 35 million TB this year rising past 100 million TB next year — an increment roughly equal to adding another entire smartphone market.

6. An assembler’s margin doubles, and Google’s three numbers.

A server assembler originally guided gross margin at 8.2% to 8.4%, then revised to 15% to 17%. When a company doing assembly work doubles its margin, the episode offers three candidate explanations: higher content value per unit in the new generation of designs, complexity making the work harder to take on and therefore raising bargaining power, and yield learning curves finally arriving. Peer share prices moved together, meaning the market read it as a condition of the whole segment rather than one company’s story.

Google contributed three numbers: API token throughput rose from 16 billion to 22.1 billion per minute, about 37% in one quarter; cloud backlog continued growing off an already high base; capital expenditure was revised up to roughly 205 billion US dollars. At the same time, free cash flow turned negative for the quarter. Separately, its in-house silicon booked external revenue for the first time — only a small portion recognized so far.

The host’s read on that last item is intuitive: renting cloud service is collecting rent — build the chip, build the server, then recover the money year by year. Selling the chip outright is selling the house — you get the cash back at once. Same investment, different cash recovery speed, therefore different sustainable cycle length.

Extended Thoughts

1. To tell noise from structure, ask whether the total or the mix is changing

A headline like “model cuts memory by 75%” invites an immediate, sweeping conclusion: so we don’t need to buy as much memory?

Take it apart and the answer is no. The saving applies to one specific block. The extra tokens spent detouring eat another block right back. And tiered offloading shifts demand from the most expensive layer to a cheaper one — it doesn’t make demand vanish. Put together, what changed is the mix and the location, not the total.

That gives a reusable test. When you meet a claim that some technology makes some component unnecessary, first ask whether it changes the total or the mix. Total changes are structural. Mix-only changes are noise — except to whoever’s share within that mix is growing or shrinking, for whom they’re entirely real. The same news meaning opposite things at different layers is normal. The error is reading it as one directional story.

That said, this test has its own failure condition worth naming: if a shift in mix persists long enough at large enough scale, it becomes a question of totals on its own. Moving from the expensive layer to the cheap one is substitution in the short run; over time it changes each layer’s capacity investment decisions, which loops back into totals. So this test is for blocking the first-reaction overreaction, not for permanent classification.

2. Bottlenecks move, and the direction of movement is derivable

This episode is effectively a field report on where the bottleneck currently sits. Compute is tight, so attention shifts to interconnect. Interconnect costs rise, so someone compensates with more pins. Memory is tight, so the cache gets tiered downward. Power density is tight, so 800-volt distribution architectures get pushed.

What’s worth noticing is that each relief doesn’t eliminate the bottleneck, it pushes it one station down the line. Decoupling training from a proprietary numeric format moves bargaining power from silicon toward the format. Offloading cache to flash moves pressure off the most congested process node onto a different one. Scaling superpods up costs you switching and topology difficulty. Trace the chain and you can broadly reason out where the next tight spot lands.

This also yields a failure condition worth writing down: if a layer’s substitution path genuinely works, that layer’s pricing power gets diluted. That statement is testable — not by feel, but by watching actual adoption of the substitute: is it still spec sheets and keynote slides, or is it deployed at scale? The episode is honest about this, describing these approaches as “in progress, but genuine large-scale deployment probably needs more observation.” That honesty is worth more than the conclusion, because it separates “has happened” from “is being discussed.”

3. Leading indicators, lagging indicators, and adjusting only one side of the ratio

The back half of the episode spends real time on a debate: when does capital expenditure peak? The mainstream answer is somewhere in the second half of next year through the first half of the year after, reasoning from free cash flow — once you’ve invested to the point where cash flow turns negative, there isn’t much headroom to add more.

The host’s response is the key move: free cash flow is a lagging indicator. What you can compute now is what everyone can already see. Genuine leading indicators live elsewhere — token consumption, actual application adoption, and the credit market.

The credit market segment is the most methodologically useful part of the episode. They note that since June, spreads on these large cloud companies’ bonds have been widening. Wider spreads mean higher borrowing cost, and borrowing cost is a component of the discount rate — a bigger denominator should, in theory, push valuations down. The lesson from 2008 is that credit default swaps were bought up first while equities caught on later; bond markets have a different participant mix and different information sources, so they tend to react earlier. Beyond the information advantage there’s a more mechanical reason: when bonds fall hard, allocation-driven capital has to sell equities to rebalance. That selling pressure has nothing to do with fundamentals, but it’s real.

Let me close this section on a discipline point: the numerator and the denominator move at the same time, and adjusting only one guarantees you’ll arrive at the answer you wanted. This episode handed us signals on both sides. Numerator: token volume up 37%, backlog still growing, assembly margins revised up. Denominator: spreads widening, cash flow turning negative. Someone watching only the numerator concludes it’s early. Someone watching only the denominator concludes it’s over. The honest approach is to adjust both and accept that the conclusion might be “I don’t know.”

As for when the game finally slows, the standard the episode offers is refreshingly plain: when usage growth stops covering spending growth — when the volume-times-price arithmetic stops working — it slows down on its own. The virtue of that formulation is that it can be tracked continuously rather than requiring a date to bet on. Rather than flipping your view every quarter on a single new number, fix “usage growth versus spending growth” as a standing comparison and check your answer against it over time.

Disclaimer

This article is a set of listening notes and extended reflections on a podcast episode. It is a personal, educational summary and does not constitute investment advice, an offer, or a solicitation. Companies, products, and technologies are mentioned solely to illustrate industry structure and should not be read as a recommendation to buy or sell any security. No price targets or entry and exit timing are provided.

Figures cited here are drawn from spoken remarks on the show and may be imprecise or out of date. Investing carries risk. Any investment decision should rest on your own financial situation, risk tolerance, and independent research, and you should consult a qualified professional where appropriate. The author accepts no liability for losses arising from use of this content.

Raw Markdown

This article is an educational discussion of investment method. It is not advice to buy or sell any individual security, offers no target prices, and does not analyze any current holding. Investing carries risk; make your own decisions or consult a qualified professional.