# bluenotebook.io — full content > All posts from bluenotebook.io, the technical blog of Nikhil Kasukurthi, as plain markdown. Index at https://bluenotebook.io/llms.txt. # Does a context gate for search agents actually work? > Testing context gating on BrowseComp-Plus. A cheap label-free gate cuts a search agent's input tokens 1.4x, gate costs included, no detectable accuracy change. Published: 2026-07-17 Author: Nikhil Kasukurthi Tags: LLM, Agents, RAG, Evaluation Canonical: https://bluenotebook.io/blog/context-gating-browsecomp/ --- At Berlin Buzzwords 2026, Lester Solbakken gave a talk titled "Better retrieval makes agents worse" [+buzzwords]. These two slides matched my experience building search for agents.
Slide from the Berlin Buzzwords talk: First Drop of Ink, a small fraction of hard distractors causes disproportionately severe, nonlinear performance degradation
Distractors degrade accuracy nonlinearly
Slide from the Berlin Buzzwords talk: agentic retrieval is context admission control. Control context admission, not just top-k. Search broadly, verify carefully, inject narrowly
Control context admission, not just top-k
Lester's argument was that agentic retrieval is **context admission control**. An agent retrieves context to act. A false positive is not a wasted result on a page: it enters the context, gets re-read at every later step, and shapes the next action. Classic retrieval optimizes recall. A search tool inside an agent has to optimize precision. The talk inspired me to build a context gate inside the search tool: a second model call that decides, per retrieved document, what enters the agent's context. BrowseComp-Plus [+browsecomp] was the test-bed. It ships with labeled evidence documents for every query, which makes the idea measurable. The rest of this post measures it: six agent-and-gate configurations over the same 180 queries. Counting its own tokens, a cheap gate cuts input 1.4×, at accuracy indistinguishable from the ungated baseline. But on DeepSeek's cache pricing the gate loses money, and it doubles per-query latency. ## Why context accumulates An agent with a search tool issues several search calls for a single question, and every result stays in the conversation until the final answer. By search 8, the agent is re-reading the distractors from search 1 on every step. Massive context windows make context stuffing the easy way out: leave everything in and trust the model to figure out the answer. It mostly works, and it pays for that in tokens and latency at every step. Prior work has measured what it also costs in accuracy. The first-drop-of-ink paper Lester cites is worth a closer read. From Gao et al. Left: accuracy against the share of hard distractors in that context. Right: the shaded first 10%, zoomed in. Of the 25 points the model loses in total, 14.5 are gone before the context is even a tenth distractors. Gao et al. [+firstink] pin the damage on hard distractors, documents close enough to the topic to pass for evidence. The first such documents to enter the context do most of the harm. In my ungated baseline run on BrowseComp-Plus, a single question costs the deepseek-v4-pro agent just over a million input tokens this way, 1,035K on average. The weaker flash agent averages 1.29M. Either agent writes roughly 1/100 of what it reads. An idealized picture of what the model re-reads at each step. Hover a row for the breakdown. Blue blocks are the evidence the answer needs. The bar grows every search, but the evidence share doesn't: by search 18, 78% of the prompt is dead weight. The model has read ~0.9M tokens to obtain three useful documents. ## The context gate The context gate is another agent/LLM call inside the `search` tool, one batched model call per search. Conditioned on the question, the gate assigns each retrieved document one of three admissions. 1. **Full document.** The complete document, truncated at a maximum token boundary. 2. **Span.** The gate returns only the snippets relevant to the question. 3. **Withhold.** The gate judges the document a distractor and returns only its title, with a note that it withheld the rest. Per-document admissions. A withheld doc keeps its title and a get_document hook, so the agent can overrule the gate. Over a full run the admissions compound: net of the gate's own reads, the gated run uses 1.4× fewer input tokens and lands within one query of the ungated baseline. The gate is write-once. It decides what enters at retrieval time and never edits the conversation afterwards, because pruning earlier messages invalidates the prefix cache. What reaches the agent is only what the gate judged relevant at that moment. ## Experiment setup All the experiments are on BrowseComp-Plus: a fixed corpus of 100K passages, with queries labeled for gold documents (contain the answer) and evidence documents (needed to connect the hops). The agent is DeepSeek (v4-flash or v4-pro depending on the arm) with a `search` and a `get_document` tool, capped at 30 iterations. I am not building a better retriever here. The baseline retriever from the BrowseComp-Plus paper is hybrid BM25 + Qwen3-0.6B embeddings, at k=10. It surfaces a labeled document for about a quarter of the annotated queries, 180 in all. I call these the *firing* queries. The gate only decides what happens after retrieval surfaces something, so the experiments run on the queries where it has ground truth to act on.
The exact run configuration Selection arithmetic: of the benchmark's 830 queries, 743 had span annotations I could validate for the oracle. Hybrid retrieval surfaces a labeled document for 199 of those (27%). Removing pilot-run queries and one query that deterministically blows the context window in the oracle arm leaves 180. Retrieval is hybrid BM25 + dense with reciprocal-rank fusion at k=10. An iteration can issue several tool calls, so search counts per query run higher than 30. The same judge model (deepseek-v4-flash) scores all arms, one trajectory per query per arm, sampling temperature left at the API default. I audited by hand the runs that ended in a raw tool-call fragment instead of a final answer, and scored judge false positives as incorrect. ```bash uv run python -m search_agent.deepseek_client \ --model deepseek-v4-pro --query runs/hybrid/newfiring182.tsv \ --get-document --query-template QUERY_TEMPLATE \ --searcher-type hybrid --bm25-index-path indexes/bm25 \ --dense-index-path 'indexes/qwen3-embedding-0.6b/corpus.shard*_of_4.pkl' \ --model-name Qwen/Qwen3-Embedding-0.6B --attn-implementation sdpa \ --snippet-max-tokens 512 --k 10 --max-iterations 30 \ --num-threads 16 --max_tokens 64000 \ --gate model --gate-model deepseek-v4-flash \ --gate-withhold soft --gate-spans-only \ --output-dir runs/hybrid/newfiring182_proagent_flashgate_v3 ``` Drop the four `--gate*` flags for the ungated baselines. The gate itself is a system prompt plus about 200 lines inside the search tool. Everything is public in [this BrowseComp-Plus fork](https://github.com/Nikhil-Kasukurthi/BrowseComp-Plus/tree/public/context-gating): the gate and its prompts (the deployed one is `GATE_SYSTEM_PROMPT_SPANS_ONLY` in [`span_labeling/gate.py`](https://github.com/Nikhil-Kasukurthi/BrowseComp-Plus/blob/public/context-gating/span_labeling/gate.py)), the agent CLI, the experiment logs, and the [recompute script](https://github.com/Nikhil-Kasukurthi/BrowseComp-Plus/tree/public/context-gating/analysis/canonical_180) behind every number in this post. The fork stores query IDs only; BrowseComp's canary-protected queries regenerate locally via the upstream decrypt script.
## The experiments Before building a real gate, I tested the idea with an oracle that reads the answer key. The oracle admits only the labeled documents, with pre-annotated answer spans where they exist, and withholds everything else the search returns. With `deepseek-v4-flash` as the agent, the oracle gate scores 80% to the vanilla searcher's 79%. And it uses 2.5× fewer tokens. Its score is a ceiling on withhold precision only, not on end-to-end accuracy. Incomplete annotations make the oracle withhold documents that are actually useful, so a real gate is not bound by it. For judging the final answer, I re-use the BrowseComp-Plus judge mechanism, with deepseek-v4-flash as the judge model. The deployable version has no answer key to look up. It reads each question–document pair and predicts the admission from the text alone. Two of its design decisions came out of watching early runs fail. ### 1. Withholding must be soft. The first version replaced withheld documents with a bare `[withheld]` marker. The agent treated every marker as a mystery box and called `get_document` on nearly all of them, pushing expansions and tokens *above* the ungated baseline. Showing the title fixed it. The agent has only one question about a withheld document, whether it is worth fetching, and the title answers it in a few tokens. The agent's reaction to withheld documents. Hiding everything triggers an expansion storm. Showing the title keeps expansions rare. ### 2. Spans beat full documents. The best variant forbids admitting any document whole. Kept documents are compressed to the verbatim spans that connect them to the question. That bounds the context no matter how long the agent runs. One query in my set blows the model's 1M-token context window when the gate admits whole evidence documents. Under spans-only it completes. ## Results Net of the tokens the gate itself reads, gating cuts the pro agent's input 1.4× with no detectable accuracy change. The ungated pro agent scores 148/180 (82.2%). With `deepseek-v4-pro` as both agent and gate, it scores 147/180 (81.7%). One query apart is a tie at this sample size, where the error bars run about ±6 points. A band that wide cannot rule out a real drop of a few points either.
A paired test on the per-query outcomes Every arm answers the same 180 queries, so the arms can be compared query by query instead of through the totals. McNemar's exact test uses only the discordant queries, the ones where one arm is right and the other is wrong. | comparison | scores | discordant (first right vs second right) | exact p | |---|---|---|---| | pro ungated vs pro + pro gate | 148 vs 147 | 31 (16 vs 15) | 1.00 | | pro ungated vs pro + flash gate | 148 vs 146 | 26 (14 vs 12) | 0.85 | | flash ungated vs flash + flash gate | 143 vs 137 | 32 (19 vs 13) | 0.38 | | flash ungated vs flash + oracle gate | 143 vs 145 | 32 (15 vs 17) | 0.86 | Nothing approaches significance, including the six-query drop under the flash agent. The discordant counts are the more informative number. Any two arms disagree on 26 to 32 queries, about 17% of the set, while the net difference never exceeds six. Most of the query-level movement between arms is trajectory variance rather than anything the gate did.
All six experiments on the same 180 queries, same judge. Hover or tap a point for that arm's full numbers. Gating moves every arm left (fewer tokens). None moves up detectably. The agent's own reads fall from 1,035K to 589K tokens per query, a 1.7× cut. Adding back what the gate reads nets out at the 1.4×. Swapping the pro gate for a flash gate keeps nearly all of that reduction and costs one more query, 146/180. Under the weaker flash agent the compression is deepest. Accuracy also drops six queries, from 143/180 (79.4%) to 137/180 (76.1%). The failure happens when one needed document is hidden and the agent wanders, issuing excess search calls. ## What gating costs The API costs come from each arm's measured usage and DeepSeek's price sheet [+pricing]. ### API costs | arm | accuracy | agent $/query | gate $/query | total $/query | total, 180 queries | |---|---|---|---|---|---| | pro, ungated | 148/180 (82.2%) | $0.063 | n/a | $0.063 | $11.31 | | pro + flash gate | 146/180 (81.1%) | $0.044 | $0.030 | $0.074 | $13.36 | | pro + pro gate | 147/180 (81.7%) | $0.042 | $0.093 | $0.135 | $24.34 | | flash, ungated | 143/180 (79.4%) | $0.027 | n/a | $0.027 | $4.80 | | flash + flash gate | 137/180 (76.1%) | $0.015 | $0.026 | $0.041 | $7.31 | The ungated baseline is the cheapest arm for its accuracy, because of context caching. The ungated agent re-reads a long, stable prefix on every step, and almost all of those reads are billed as cache hits. The gate reads each document once, in a fresh prompt, so its tokens are billed at cache-miss prices. Despite the token reduction, gating costs more per query. 89% of the baseline's input tokens are cache hits. Cache-hit tokens are billed at 1/120th of the miss price. The 165M tokens of stale re-reading come to about $0.60. The tokens the gate removes were nearly free. At cache-miss prices throughout, the ungated arm would cost $82.65 and the gated one $56.06. That is a 1.47× saving. A straight line between the measured run and that no-cache hypothetical puts the crossover near an 83% cache-hit rate. This run sat at 89%. On a well-cached agent with a 99% cache discount, stale context is almost free to re-read, and a gate has to justify itself some other way. ### Latency I did not instrument detailed timing during the runs. I did keep the agent traces, with timestamps, and each gate's call counters. Every latency number below is a median, because API dropouts contaminate individual timings. On serial single-thread runs, the median query takes 83 seconds ungated and 165 seconds gated, twice as long. Per-query wall-clock on serial single-thread runs (flash agent, 14 baseline and 16 gated samples), reconstructed from run persist timestamps. Hover a band for the numbers. The flash-gate tail runs well past its band, topping out at 727 s. The oracle arm applies the same gating semantics through a zero-cost label lookup, and it is *faster* than the baseline, a median of 66 seconds per query to the baseline's 83. Admitting gold spans early ends queries in fewer searches. The mechanism is one blocking model call per search, on the critical path by construction. The agent's next turn conditions on the gated results, so there is no async escape. The gate memoizes decisions per document, but reformulated searches mostly surface new documents. In practice it fires 17–22 times per query, at 7–10 seconds a call. Those seconds are mostly decode. Spans-only means the gate writes the evidence out verbatim. That is about 800 completion tokens per call for the flash gate, 1,900 for pro. Spans-only bounded the context window, and spans-only decode is most of the added latency. ### Cutting the overhead Three levers: 1. **Emit character offsets, not verbatim spans.** The search tool already holds the document text. Let the gate return `start:end` and slice locally. That cuts the decode from ~800 tokens to under 100. Per call, an estimated 8 seconds drops to 1–2. This needs a gate model competent at emitting exact offsets. Smaller models struggle with it. 2. **Gate documents in parallel.** One batched call decodes every span serially, while k single-document calls decode only as long as the slowest one. Composes with the first lever. 3. **A purpose-built small gate model.** Distill the gate's decisions into a 0.6–8B cross-encoder-style model served next to the agent: sub-second per search, no API round-trip. The agent×gate grid already showed gate capability is not the binding constraint. The flash gate ties the pro gate under a pro agent. Ding et al. see the same shape: robustness to retrieval noise rises with agent strength [+robustrag]. ## Where the losses come from I read the trajectory of every query a gated arm lost against the baseline, with the gate decisions alongside. (One direction only: the queries the gated arms *won* have not had the same close read.) Losses against the ungated baseline, classified from the trajectories and gate decisions. Hover a bar for the query IDs. The failure mode specific to gating is **trajectory steering**. The gate withholds a document early, and the agent's later query reformulations drift away from the gold documents. On one query the baseline retrieved all three gold documents. The gated agent issued 56 search calls across its 30 iterations and retrieved none of them. A per-search gate cannot know its decision will derail the search three steps later. This affects 8 of 180 queries, 4.4%. I had been preparing the training side of this idea: distill the gate into a small open-weights model, then RL with trajectory-level reward to fix this steering. The census shelved the RL half. Steering affects 4.4% of queries, so at best RL recovers half an accuracy point. That gain sits entirely inside the ±6-point noise band. Distillation is still worth doing, but only for the 7–10 seconds per gate call. The census also showed the largest failure bucket has nothing to do with gating. The agent holds the right documents, hits the 30-iteration cap without committing to an answer, and emits a raw tool-call fragment. On the oracle arm this accounts for about 9 of its 15 losses. For accuracy on this benchmark, termination handling is worth more than any gate improvement. ## One benchmark, one retriever, one model family Everything above is measured on one setup. BrowseComp-Plus queries are multi-hop questions over a fixed 100K-passage corpus, and the 180 firing queries are the subset where the stock hybrid retriever surfaces labeled evidence. At this sample size the error bars are about ±6 points, wide enough to swallow every accuracy difference between arms. The paired McNemar test in the results section says the same thing. Agents, gates, and judge are all DeepSeek v4 models. deepseek-v4-flash scored every arm, one trajectory per query. A judge from the same family as the system it grades can share that system's blind spots, and one trajectory gives its quirks no chance to average out. A different agent family may react differently to gate-compressed context. I did not vary the gate prompt, so its sensitivity to wording is unmeasured. And nothing here says how the gate behaves on an unlabeled corpus, where no annotation ever checks its withhold decisions. ## Takeaways On this DeepSeek setup, the gate holds up as compression, 1.4× fewer input tokens with no training. The ink paper reaches the same verdict from a controlled direction. In its filtering experiments, the gain came from the shorter context rather than from which documents were removed [+firstink]. It loses on cost here because DeepSeek's cache discount makes stale context nearly free to re-read. The dollar case needs a setting where caching is weak or absent. And as built, it costs about 2× wall-clock latency, most of which the levers from the latency section would remove. I have not built them yet. ### When to use a gate - **Cache-hit rate around 83% or lower.** Below that crossover the gate saves money as well as tokens. The 83% is interpolated from one measured run, so it is approximate. Multi-turn products that interleave user messages between searches, and serving stacks without prefix caching, sit well under it. - **The context window binds before the budget does.** Spans-only admission caps what each search can add, so the context stays bounded however long the agent runs. - **A few extra seconds per search are acceptable.** Each gate call blocks for 7–10 seconds as built. Character offsets and parallel calls remove most of that, but only after you build them. - **Never for accuracy.** No configuration detectably beat the ungated baseline, and on this benchmark termination handling is worth more than any gate improvement. Code, gate prompts, experiment logs, and the analysis behind every number in this post are in the [BrowseComp-Plus fork](https://github.com/Nikhil-Kasukurthi/BrowseComp-Plus/tree/public/context-gating), branch `public/context-gating`. [+buzzwords]: Lester Solbakken. "When better retrieval makes agents worse." Berlin Buzzwords 2026. [Talk recording](https://youtu.be/07kARdSIjVI). Lester builds [Hornet.dev](https://hornet.dev). [+firstink]: Gao, Chen, and Huang. 2026. "The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning." The paper behind the talk's ink slide: a small fraction of hard distractors causes most of the degradation, and filtering gains come mainly from context-length reduction rather than distractor removal. https://arxiv.org/abs/2605.10828 [+browsecomp]: Chen et al. 2025. "BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent." ACL 2026. OpenAI's BrowseComp queries (Wei et al. 2025, https://arxiv.org/abs/2504.12516) rehosted over a fixed 100K-document corpus with labeled gold and evidence documents, indexed with BM25 and Qwen3 embeddings. https://arxiv.org/abs/2508.06600 · [dataset](https://huggingface.co/datasets/Tevatron/browsecomp-plus-corpus) [+robustrag]: Ding et al. 2025. "On the Diminishing Returns of Complex Robust RAG Training." SIGIR-AP 2025. The robustness benefit of defending an LLM against noisy retrieval shrinks substantially as model capacity grows. https://arxiv.org/abs/2502.11400 [+pricing]: DeepSeek [price sheet](https://api-docs.deepseek.com/quick_start/pricing) as of July 2026, per 1M tokens. v4-flash: $0.14 input (cache miss), $0.0028 input (cache hit), $0.28 output. v4-pro: $0.435 / $0.003625 / $0.87. Arm costs computed from each run's measured cached/uncached/output token counts. Gate tokens are priced as cache misses, which makes the gate's cost a slight upper bound. --- # Which H100 instance to train Nanochat > Benchmarking H100 PCIe vs SXM vs NVL on training cost, step times, and NCCL profiling to find the cheapest GPU configuration for Nanochat Published: 2026-03-04 Author: Nikhil Kasukurthi Tags: GPUs, LLM, Training, Nanochat Canonical: https://bluenotebook.io/blog/h100-nanochat-training/ --- Training to GPT-2 level performance on CORE metric [+recent_benchmark] has dropped from $43K in 2019 to $73 in 2026. I wanted to train Nanochat [+nanochat] on spot instances, where Karpathy mentions the cost can be even lower to $20 on 8xH100 GPUs. But, on Runpod, I was confronted with a choice - H100 PCIe, SXM or NVL. Each at varying price points.
Choice of H100 - PCIe, SXM, NVL
Runpod offers H100 in multiple configurations
I knew these were different network interconnect options from the CS336 course [+cs336] and that NVLink 4.0 was supposed to be fast. Prof. Percy Liang mentions in the first lecture of CS336, the mindset while training LLMs is to squeeze most performance of the hardware. This mindset from the course, led me to examine what each of these interconnect variants has to offer. To train the model cheaply, is the cheapest instance the best choice to complete the training run? I decided to benchmark all three. [+nanochat]: "The best ChatGPT that $100 can buy." [Nanochat Repository](https://github.com/karpathy/nanochat/tree/master) [+recent_benchmark]: https://github.com/karpathy/nanochat/discussions/481 [+cs336]: lecture 5 https://www.youtube.com/watch?v=6OBtO9niT00 ## TL;DR I benchmarked 8xH100 on SXM, PCIe and NVL offerings across Runpod and Vast.ai SXM, although expensive per hour, is the clear choice to train Nanochat within 3 hours at ~$37. Making it 2x cheaper than PCIe and 3x cheaper than NVL. But, even SXM can regress if GPUs are split across NUMA nodes. ## Why care about the network interconnect? While training on multiple GPUs, most parallelism techniques use the interconnect to transfer gradients at every step. The current implementation of Nanochat takes about 3 hours to train on an 8xH100. The optimizer is the only distributed component. Nanochat uses a combined Muon + AdamW optimizer [+DistMuonAdamW]. Muon handles all large 2D matrices[+muon_ops], the transformer block essentially. AdamW for the rest: input token embeddings (wte), LM head, Value embeddings, and two small residual addition scaling parameters (x0_params and resid_params). The optimizer runs in two stages: phase 1 is for reduce ops and phase 2 is for gather ops. [+muon_ops]: Muon handles - Attention projections (Q, K, V, O) and MLP weights (c_fc, c_proj) plus the tiny Value Embedding Gates. Phase 1 averages gradients across devices[+devices]. `all_reduce` and `reduce_scatter` primitives are used for this. In Nanochat, `all_reduce` is used for tiny parameters (under 1024 elements), each rank receives the full averaged gradient in a single collective. Since these are just a few KB, the overhead to send them to all ranks is negligible. `reduce_scatter` the sharded alternative, handles rest of the parameters, each GPU receives 1/8 of the averaged gradient. ``` Phase 1 reduce_scatter(grads) GPU 0 → avg_grad[0:N/8] GPU 1 → avg_grad[N/8:2N/8] ... ```
NCCL AllGather collective operation diagram
All gather
NCCL ReduceScatter collective operation diagram
Reduce Scatter
NCCL Gather collective operation diagram
Gather
Then in phase 2, each rank runs the optimizer on its shard in isolation, producing updated parameters for that slice. After this, `all_gather` lets every rank collect all the shards, so each rank has the full updated parameter tensor for next forward pass. ``` Phase 2 optimizer(shard) → updated params all_gather(params) GPU 0 → params[0:N/8] GPU 1 → params[N/8:2N/8] ... → all ranks get full params[0:N] ``` This is the Zero-2 [+zero_2] pattern. Each GPU only needs optimizer state (momentum, variance buffers) for its shard, cutting memory to 1/world_size[+world_size]. I strongly recommend watching Lecture 7 [+lecture7] in CS336 to get a deeper idea. All figures above are from Nvidia's [NCCL documentation](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html), which has great visualisation to understand this. ## Analytical estimates of the data transfer for d26 Nanochat From the Nanochat model architecture, we can estimate the data transfer required for each parameter group. The optimizer moves data in two phases: ReduceScatter to average gradients, then AllGather to distribute updated parameters. Each optimizer step transfers roughly 7.1 GB across the interconnect. ~3.6 GB in AllGather (bf16), ~3.6 GB in ReduceScatter (split between bf16 and f32), and a negligible AllReduce for the two small lambda parameters. We get this value by adding the tensor sizes across all parameter groups - lm_head, wte, value_embeds, and the Muon-managed transformer blocks.
Per-group communication volume & NCCL op summary (per optimizer step) | group | kind | num_params | padded_count | elements_per_param | total_elements | RS (MB) | AG (MB) | AR (MB) | |:------------------|:-------|-------------:|---------------:|:---------------------|:-----------------|----------:|----------:|----------:| | lm_head | adamw | 1 | 1 | 54,525,952 | 54,525,952 | 109.1 | 109.1 | 0 | | wte | adamw | 1 | 1 | 54,525,952 | 54,525,952 | 109.1 | 109.1 | 0 | | value_embeds | adamw | 13 | 13 | 54,525,952 | 708,837,376 | 1417.7 | 1417.7 | 0 | | resid_lambdas | adamw | 1 | 1 | 26 | 26 | 0 | 0 | 0 | | x0_lambdas | adamw | 1 | 1 | 26 | 26 | 0 | 0 | 0 | | muon (13, 32) | muon | 13 | 16 | 416 | 6,656 | 0 | 0 | 0 | | muon (1664, 1664) | muon | 104 | 104 | 2,768,896 | 287,965,184 | 575.9 | 575.9 | 0 | | muon (1664, 6656) | muon | 26 | 32 | 11,075,584 | 354,418,688 | 708.8 | 708.8 | 0 | | muon (6656, 1664) | muon | 26 | 32 | 11,075,584 | 354,418,688 | 708.8 | 708.8 | 0 | NCCL op summary per step (compare with nsys CUDA GPU Kernel Summary) | nccl_op | dtype | calls_per_step | total_MB | avg_MB_per_call | min_MB_per_call | max_MB_per_call | |:--------------|:--------|-----------------:|-----------:|------------------:|------------------:|------------------:| | AllGather | bf16 | 19 | 3629.4 | 191 | 0 | 708.8 | | AllReduce | bf16 | 2 | 0 | 0 | 0 | 0 | | ReduceScatter | bf16 | 15 | 1635.8 | 109.1 | 109.1 | 109.1 | | ReduceScatter | f32 | 4 | 1993.6 | 498.4 | 0 | 708.8 |
That 7.1 GB is the tax every single training step pays. ## Choice of H100 Back to our first question, which H100 instance to choose? Most providers offer the H100 in two form factors SXM and NVL [+h100_offering]. SXM variant is a custom baseboard from Nvidia, whereas NVL is installed through the PCIe dual-slot. The Hopper architecture introduced FP8 which also led to the faster training time on the Nanochat leaderboard. At that precision SXM delivers 3,958 TFLOPS vs NVL's 3,341. These GPUs can be interconnected through NVLink or PCIe. NVLink offers 900 GB/s (bidirectional) on SXM vs 600 GB/s on NVL, and just 128 GB/s on PCIe. One important note is that on NVL instances, only two GPUs can be connected through NVLink. Within a pair, NVLink on NVL gives 300 GB/s per direction, and cross-pair traffic falls back to PCIe. On SXM instances, NVSwitch connects all GPUs in a mesh providing 450 GB/s per direction (900 GB/s bidirectional).
H100 SXM Systems View
H100 SXM Systems View showing the GPU interconnect. Figure from Hopper [white paper](https://docs.nvidia.com/enterprise-reference-architectures/white-paper.pdf#page=20.20)
H100 NVL Systems View and Network Topology
H100 NVL Systems View and Network Topology. Figure from Hopper [white paper](https://docs.nvidia.com/enterprise-reference-architectures/white-paper.pdf#page=20.20) and H100 NVL [product brief](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/h100/PB-11773-001_v01.pdf#page=14.10)
They also differ in max thermal design power (TDP). SXM can go up to 700W while NVL peaks at 400W. Higher power draw unlocks better clock speeds and, in turn, better FLOPs per dollar. Horace He explores this in a fun blog[+horace_he] about how data values affect power draw. Predictable data - complete zeros or ones flip fewer transistors, leading to less dynamic power and in turn better clock speeds. [+horace_he]: https://www.thonking.ai/p/strangely-matrix-multiplications Runpod is one of the few providers to offer all three - SXM, NVL and PCIe. The cost for SXM is higher than PCIe but cheaper compared to NVL. Vast.ai also has SXM configuration at varied price points for both on-demand and spot instances, in many cases, cheaper than Runpod's SXM. Prices as of early March 2026, US regions. Cloud GPU pricing changes frequently. | | Runpod PCIe | Runpod NVL | Vast.ai SXM | Runpod SXM | |---|---|---|---| --- | | 8-GPU node/hr (on-demand) | $19.12 | $21.52 | $12.85 | $21.5 | | 8-GPU node/hr (spot) | $10 | $13.2 | $7-$10 | $14 | The theoretical bandwidth ratio between NVLink (~450 GB/s per direction on SXM) and PCIe 5.0 (~64 GB/s per direction) is roughly 7x [+h100_offering]. If that ratio holds in practice, SXM should recoup its price premium on interconnect savings alone. [+h100_offering]: [Nvidia H100 specification](https://www.nvidia.com/en-in/data-center/h100/) ## Benchmarks My initial hypothesis was SXM instances were more expensive per hour, but they would be cheaper to complete the training run. From the nanochat leaderboard[+leaderboard], the d26 GPT-2 record uses `--target-param-data-ratio=8.5` with FP8, training on ~7.8B tokens at batch size 524,288 for 14,889 steps to reach CORE 0.2578 (original GPT-2: 0.2565). Each step has the model forward, backward and the optimizer step. [+leaderboard]: d26 + FP8 [link](https://github.com/karpathy/nanochat/blob/master/dev/LEADERBOARD.md) My first benchmark used PCIe (starting with the cheapest option) which gave a baseline of ~1400ms. Next, when I ran the with SXM, it was just a minor improvement to ~1200ms. PCIe would be a better choice to complete the training run at this level. These results were starkly different to my hypothesis. The first difference between the two instances was PCIe had 252 vCPUs and SXM only 160 vCPUs. Then I found a Vast.ai offering for 256 vCPUs for SXM, which outperformed PCIe at 700ms. Finally in the promised land of performance. To check if this improvement was purely due to vCPU sizing, I ran with 128 vCPUs SXM on vast.ai and found it match the 256 vCPUs SXM run. In later sections, I detail about this disparity and the possible causes in the apparent regression of SXM 160 vCPUs on Runpod. And finally, I benchmarked the NVL configuration for completeness. In the results reported, I only talk about the three variants - SXM 128 vCPUs (on vast.ai), PCIe 252 vCPUs and NVL 128 vCPUs on Runpod. The experiments that failed are in the final section. I wrote this profiling script [here](https://github.com/Nikhil-Kasukurthi/nanochat/blob/11dab2952bb64fe6b0d1347b1f1e5ca71dcac908/scripts/profile_comms.py) [+profile_comms.py] which performs a warmup of 3 steps and then profiles 10 steps. I use `torch.cuda.Event` to time each step. This also isolates the optimizer's average time, revealing network overhead. I measured compute and network time separately, even though they overlap during normal training. The measured times will be slightly higher than actual Nanochat training. Along with this, I also added Nvidia's `nsys` to tool annotate specific parts of the script. Through `torch.cuda.nvtx.range_push` each operation's timing is broken down. The `nvtx` ranges and cuda events are split into three phases - Phase 1-Reduces, Phase 2-Compute+Gather, Phase 3-WaitGathers. Phases 1 and 2 are GPU-intensive. They perform network collectives and fused optimizer kernels. Phase 3 is a synchronization step where CPU waits on network completion. ### Measured Step Times for d26 Profiled at device_batch_size=32, total_batch_size=524,288 (no gradient accumulation). The d26 GPT-2 record (Run 2) uses the same batch size with device_batch_size=16 and grad_accum=2, which produces equivalent step times. SXM completes each step in ~702ms. Nearly half the time of PCIe, and a third of NVL. | Platform | vCPUs | Avg Step Time | Optimizer Step | Comm Overhead | Relative | Training Time | |----------|-------|--------------|----------------|---------------|----------|---------------| | SXM (NVSwitch) | 128 | **701.9 ms** | 57.8 ms | 8.2% | 1.00x | **2.90 hours** | | PCIe | 252 | 1411.6 ms | 375 ms | 26.6% | 2.01x | 5.84 hours | | NVL | 128 | 2031.5 ms | 395.6 ms | 19.5% | 2.89x | 8.40 hours | From the earlier section, we know SXM's NVSwitch mesh gives every GPU full bandwidth to every other GPU. PCIe is limited to ~64 GB/s per direction, and NVL only has NVLink within pairs, cross-pair traffic falls back to PCIe. The results echo the same. ### NCCL Communication Measured GPU kernel execution times from Nsight Systems. All three runs produced the same total kernel call counts, enabling direct comparison of total times. [+profile_comms.py]: [profile_comms.py](https://github.com/Nikhil-Kasukurthi/nanochat/blob/11dab2952bb64fe6b0d1347b1f1e5ca71dcac908/scripts/profile_comms.py) The measured 7.3x difference in total NCCL kernel time between SXM and PCIe lines up almost exactly with the spec sheet's 7x bandwidth ratio. ### Per-Kernel Average Latency From nsight, I exported the NCCL calls from the CUDA GPU Kernel Summary across all configurations. SXM performs the best here. NVL has NCCL kernel times nearly identical to PCIe. NVL step time (2031 ms) is 44% worse than PCIe (1412 ms) even though NCCL kernel times are nearly identical. One plausible explanation: on NVL, inter-pair traffic shares the PCIe bus with host-to-device transfers, starving both. However, I did not verify this with `nvidia-smi topo -m` on the NVL node, nor did I check whether the NVL instance had a NUMA split similar to the SXM regression described later. This anomaly deserves deeper investigation. ## Model Size Sensitivity (d12 vs d26) Does a smaller model show the same interconnect sensitivity? I profiled d12 (286M params, device_batch_size=32, grad_accum=1) alongside d26 for two configurations. Smaller models are more communication-sensitive because their compute-to-communication ratio is lower — less time in matmuls means the interconnect bottleneck becomes a larger fraction of each step. d12 spends 23% of step time in communication on SXM vs d26's 8.2%. Surprisingly, Phase 1 time of NVL is faster than SXM for d12 likely because the small reduce volume fits within a single NVLink pair's bandwidth, avoiding NVSwitch overhead.
d12 Optimizer Phase Breakdown | Platform | Phase 1 | Phase 2 | Phase 3 | Total Optimizer | |----------|---------|---------|---------|-----------------| | SXM 128 vCPU | 4.2 ms | 25.2 ms | 11.7 ms | 41.2 ms | | NVL 128 vCPU | 2.7 ms | 36.3 ms | 23.4 ms | 62.3 ms |
## Takeaways For a total of 14,889 steps, SXM completes the training in nearly half the time of PCIe and a third of NVL. At $12.85/hr on Vast.ai, it's also the cheapest per-hour option. | Provider | Config | $/hr (8-GPU) | vCPUs | Step Time | Training Cost (projected) | |---|---|---|---|---|---| | Vast.ai | SXM | **$12.85** | 128 | **701.9 ms** | **$37.27** | | Runpod | PCIe | $19.12 | 252 | 1411.6 ms | $111.66 | | Runpod | NVL | $21.52 | 128 | 2031.5 ms | $180.77 | These projected costs assume linear scaling (hourly rate × training hours) and do not account for environment setup, compilation, data loading, or potential spot instance preemptions requiring restarts. Actual costs will be somewhat higher depending on the startup scripts. SXM configurations seem to be the norm from most providers. Runpod was the only provider to have all three configurations and also offered them as spot instances. Vast.ai is a bit of a lucky draw essentially, since it's a marketplace not all the configurations are available consistently. For shorter training runs like Nanochat, it is the best fit. Lambda.ai has 208 vCPU count and offers only SXM, Modal also offers only SXM and has a configurable CPU count since they are serverless. Through this exercise, I now have a better intuition on how to train with spot instances. I wrote this [`profile_comms.sh`](https://github.com/Nikhil-Kasukurthi/nanochat/blob/master/runs/profile_comms.sh) script to fail fast. It runs three checks before installing or downloading anything 1. CUDA sanity check calls `nvidia-smi` and `torch.cuda.init()` to catch driver mismatches or broken GPU state early. 2. NCCL communication check runs a dummy `torchrun` across all GPUs (`nccl_check.py`) to verify inter-GPU communication works. This catches SHM bugs and misconfigured network interfaces before any real profiling begins. 3. NUMA topology dump logs the full GPU-to-NUMA-node mapping via `nvidia-smi topo -m` and PCI sysfs lookups, so you can immediately spot a NUMA split without manual debugging. 4. Additionally, keeping an eye on the internet download speed before starting the instance is a good check. Vast.ai has some instances that are quite slow, this increases the cost of the training run since the GPUs sit idle. Only after these pass does the script install dependencies, download data, and run the actual `nsys` profiling for d12 and d26. ## Mistakes I made and issues I ran into ### 1. CPU starvation on the SXM run and NUMA socket pinning My benchmark on SXM with 160vCPUs on Runpod clocked 1295ms per step, barely faster than PCIe's 1412ms with 252 vCPUs. With higher FLOPs and faster interconnect SXM should have been a massive step-up, not a minor improvement. Assuming it's CPU starvation, I found a 256 vCPUs instance on vast.ai and got 702ms, 2x improvement. Through Nsight Systems, I found the GPU kernels themselves were fast, but they spent long stretches idle, waiting for the CPU to signal the next chunk in NCCL's ring protocol. The `pthread_cond_signal` count was 1.58 million in a 10-step profile on the 160vCPUs, vs ~4,000 on a healthy instance with 256 vCPUs. Running more experiments on Runpod with NVL and PCIe, I ran into multiple issues - slow internet on the VM, CUDA driver issues and also NCCL misconfigurations.
SXM 160 vCPU NUMA-split Nsight timeline
SXM 160 vCPU (NUMA-split). The OS runtime row is dense with syscalls.
SXM 256 vCPU Nsight timeline
SXM 256 vCPU. The OS runtime row is empty.
H100 PCIe Nsight Systems timeline
PCIe instance.
To ensure I was not fitting data to my narrative, I re-ran on Vast.ai with 128 vCPUs and got 701.9ms. Identical to the SXM with 256 vCPUs. The CPU was not the only bottleneck. Dumping the machine topology revealed the most striking difference: Runpod split GPUs 4+4 across two NUMA[+numa] nodes, while Vast.ai placed all 8 on NUMA node 0. There were also CUDA driver version differences (560 vs 570) and different kernel configs between the two hosts, so I can't attribute the regression to a single cause. That said, the NUMA split is the strongest hypothesis, and here's why. Multi-socket[+socket] servers have a NUMA (Non-Uniform Memory Access) architecture, each CPU socket has its own local memory. Accessing local memory takes ~10ns, but reaching memory on the other socket crosses the UPI (Ultra Path Interconnect) at ~100ns. When GPUs are split across NUMA nodes, NCCL's CPU-side coordination threads, the ones signaling `pthread_cond_signal` and holding mutexes, pay this cross-socket penalty on every ring protocol step. The OS scheduler makes it worse: without explicit pinning, it can schedule a thread managing GPU 5 (socket 1) onto a core on socket 0, turning every memory access and signal delivery into a cross-UPI hop. `numactl --cpunodebind=N --membind=N` pins processes to a specific socket, but NCCL spawns its own internal threads which may not respect this. The clean fix is what Vast.ai had: all 8 GPUs on a single NUMA node, so cross-socket latency never enters the picture. GPU-to-GPU NVLink communication is unaffected by NUMA since the bits travel over NVSwitch (data plane), never touching the CPU. But NCCL's control threads, which orchestrate these transfers, run on the CPU (control plane). There is active discussion on PyTorch to include NUMA pinning to `torchrun` - [Link](https://github.com/pytorch/pytorch/issues/148689). I haven't isolated whether NUMA, drivers, or kernel config dominates. I'll cover it in a follow-up post with controlled experiments. > Run `nvidia-smi topo -m` on every new instance before benchmarking. If GPUs span multiple NUMA nodes, expect NCCL overhead. And always profile before trusting step times, a bad instance can masquerade as "SXM isn't worth it." [+numa]: Non-Uniform Memory Access is a memory layout design used in data center machines. [Link](https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html#utilize-non-uniform-memory-access-numa-controls) [+socket]: A CPU socket is the physical connector on the motherboard that holds one CPU chip. A dual-socket server has two CPUs, each with its own local memory and PCIe lanes. ### 2. Spot instances being preempted mid-profile Spot instances are 30 to 50% cheaper than on-demand instances. But the trade-off is they can be shut down at any point with a 5-second notice. Since the profiling takes roughly 12 minutes including installation, env setup and actual profiling, I was confident I could get the work done on spot instances. But I did run into shutdowns a couple of times. ### 3. Broken Nodes throwing CUDA errors I ran into this issue a few times, where the host has not been configured correctly. Likely CUDA driver or GPU state was broken due to driver mismatch. Fixing this issue on the pod that is billed by the second is expensive. Shutting it down and trying at a later time is the best alternative. ``` >>> import torch, sys, os >>> >>> print(f'PyTorch {torch.__version__}, built with CUDA {torch.version.cuda}') PyTorch 2.8.0+cu128, built with CUDA 12.8 >>> >>> torch.cuda.init() Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 379, in init _lazy_init() File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 412, in _lazy_init torch._C._cuda_init() RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. ``` ### 4. NCCL connection issues on NVL On one of the community instances of 8xH100 NVL on Runpod, there was a NCCL communication issue. The instance was unable to use SHM (Shared Memory), a fast inter-process transport using /dev/shm. I tried benchmarking anyway by disabling SHM through `NCCL_SHM_DISABLE=1`. NCCL selects a transport for each GPU pair based on what the hardware supports. On an 8-GPU NVL node, only 4 of the 28 GPU pairs share NVLink, those pairs use the NVLink transport directly. For the remaining 24 pairs, NCCL tries P2P over PCIe first; when direct P2P isn't available (common in multi-root PCIe topologies), it falls back to SHM (shared memory via `/dev/shm`), which copies data through host memory as an inter-process transport [+NCCL_details]. Disabling SHM with `NCCL_SHM_DISABLE=1` removes this fallback, forcing those 24 pairs onto IP sockets, which are orders of magnitude slower. Since most ring hops on an NVL node cross non-NVLink pairs, this effectively cripples the majority of the communication path. [+NCCL_details]: NCCL transport selection hierarchy: NVLink → P2P (PCIe) → SHM (host memory IPC) → NET (sockets). See the [NCCL source](https://github.com/NVIDIA/nccl) and this [paper](https://arxiv.org/abs/2507.04786v1) for details. So, I had to do another run on the secure cloud of Runpod which also had a NVL instance. This performed much better. | Metric | NVL (128 vCPUs) | NVL no SHM (152 vCPUs) | Degradation | |--------|---------------------|----------------------|-------------| | Step time | 2031.5 ms | 6495.1 ms | **3.2x** | | Optimizer step | 395.6 ms | 5402.9 ms | **13.7x** | | Comm overhead | 19.5% | 83.2% | — | | Total NCCL (10 steps) | 30.08s | 430.58s | **14.3x** | | AllGather avg | 8.715 ms | 142.656 ms | **16.4x** | | RS f32 avg | 30.937 ms | 393.702 ms | **12.7x** | | AllGather max | — | **1040 ms** | — | NCCL degrades by 14.3x when SHM is disabled. Note: the no-SHM instance had slightly more vCPUs (152 vs 128), which should have helped, making the SHM effect even more dramatic than the raw numbers suggest. [+lecture7]: [Lecture 7 - CS336](https://youtu.be/l1RJcDjzK8M) [+devices]: Think of devices as a host having multiple GPUs. Each GPU has a rank (its ID) [+world_size]: world_size - Total number of GPUs [+data_loader]: [dataloader.py](https://github.com/karpathy/nanochat/blob/master/nanochat/dataloader.py#L46-L70) [+zero_2]: [Zero Stage 2 Paper](https://arxiv.org/pdf/1910.02054) [+all_gather]: A mechanism to get tensors from different GPUs onto a single node [torch.distributed.all_gather](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather) [+DistMuonAdamW]: DistMuonAdamW [optim.py](https://github.com/karpathy/nanochat/blob/c7ba25214276d165eeefca7cb2060587975db189/nanochat/optim.py#L297) --- # Why Model Context Protocol (MCP)? > Enabling endless capabilities for LLMs Published: 2025-04-02 Author: Nikhil Kasukurthi Tags: MCP, LLM, Protocol, AI Canonical: https://bluenotebook.io/blog/why-mcp/ --- When using LLMs, have you ever felt knee capped, such an intelligent technology, but it doesn't have some basic functionality. Want to use the browser through the LLM itself? Well, wait for the big LLM client overlords to build it. MCP is a way to solve this, creating a unified interface for you extend your LLM client's capabilities. ## What is Model Context Protocol? One of the first things in the Model Context Protocol's (MCP) specification is this analogy: > MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools. Another popular parallel that can be drawn is to the Language Server Protocol (LSP). LSP is another similar protocol like MCP. Since I started programming only in 2012, fortunately or unfortunately, I never knew about LSP until I was setting up the Zed editor for a Python codebase, and you need to install Pylance for it to discover your code structure. Currently, it's a given that any IDE will immediately begin to understand a codebase as soon as you boot it up. In the background, this is done by the LSP server. It's a way for the IDE to understand the structure of the code and index the right things so that all of it can be searched. In the early days of the IDE explosion, every IDE had its own way of creating a program tree. LSP unified that, and well, in the era of vibe coding, it's taken for granted. Like the LSP, MCP is a specification on the interaction contract between an MCP server and an LLM client to provide it with additional tools, resources, and prompts on demand.
In the images above, you can see two tools available inside Claude desktop client, cursor and also windsurf. These are open-source tools that any supporting MCP client can use. So, extending an LLM client's capabilities is now trivial. In this blog, I make the case for why MCP should even exist. ## Why MCP? After skimming through the MCP specification for the first time or seeing MCP on X, the most natural questions are along these lines: > Considering all the popular providers and open-source models support tool calling, why is MCP even needed? And if LLMs are so smart, why can't they just read API documentation and use it? This also reminds me of the infamous comment on HackerNews when Dropbox launched -- arguing you could very easily DIY what Dropbox was trying to solve. Dropbox launch comment from Hacker News showing skepticism about the product But we know the history: Dropbox became one of the largest consumer brands. Similarly, MCP has massive potential. MCP defines a common way to enable LLM clients with more capabilities beyond the out-of-box capabilities like reasoning that recent LLMs have had. In case you want to enable your LLM client to interact and control your browser, earlier you had to use the developer APIs to build this yourself and in the process also pay for the tokens being used. I have had the Claude subscription for a while now, and through MCP, the LLM can choose to invoke my browser tool. Without needing to interact with any LLM APIs, my Claude client can now control my browser. This way, MCP enables multiple new capabilities. OpenAI has a way to provide OpenAPI schema specs to ChatGPT through GPT Actions to enable additional tools right inside ChatGPT. But the adoption of it seems limited, and it's quite deeply buried in the documentation. And now, even OpenAI has announced support for MCP! > "people love MCP and we are excited to add support across our products. available today in the agents SDK and support for chatgpt desktop app + responses api coming soon!" — Sam Altman And it also seems like Google will also hop on board soon! > "To MCP or not to MCP, that's the question. Lmk in comments" — Sundar Pichai This is a massive win for the entire ecosystem since OpenAI SDKs have usually become the industry standard. The protocol specification by MCP means any client adhering to the protocol could start using the tools with zero overhead. This unlocks an entire marketplace of capabilities. ### Shortcomings of OpenAPI/Swagger for LLM integrations OpenAPI documentation is a listing of all the possible APIs that are present and the parameters that are needed. It has a semantic understanding gap, about when and why to use them. If LLMs were to directly use OpenAPI docs, they will also lack context on what each of the parameters/arguments would need. OpenAPI definitions are typically static, so changes during runtime will not be possible. If changes are also done, broadcasting them to LLMs will be tricky again. --- Now, to drive this point home, let's look at how MCP will make an outsized impact on the entire ecosystem. ## 1. MCP is a great way to bifurcate the capability vs intelligence The first wave of AI proliferation was building a ChatGPT wrapper for everything. The second wave will not be built on ChatGPT for X, but rather X extended into ChatGPT itself. Here, I'm using ChatGPT as the defacto example since everyone has used it. In the first wave of consumer AI applications, the common theme was asking a purpose-built chatbot to perform random tasks or jailbreak its system prompt. Got a car sales chatbot? Well, it can also write you code. The jailbreaks will always happen no matter what. When the user is clearly aware of the context in which the conversation is initiated, the tool + intent gives a lot better control to application developers to focus on that experience instead of building LLM clients. This bifurcation also respects the user's intelligence while ensuring the capabilities of LLMs are leveraged to solve different issues. The worst offenders are WhatsApp-integrated booking bots that ask a million questions in a painstakingly slow manner. Filling a form through chat is the worst. An LLM interpreting natural language can help in that. Given the CRUD tools to an LLM, it can understand things like the first appointment day after. Unlike a form-filling bot. These UX improvements will help in the longer run, saving users time. ## 2. The case against building LLM chat clients At the time of writing, there were 400 million active users on ChatGPT. Of those, a majority are not developers. This means, using the LLM APIs directly is not an option for most. ### "MCP is just tools with additional steps" - Wrong Tool capabilities with LLM APIs have been around for more than a year and half now. OpenAI was the first to release these. But, you still need to write an LLM client to interface with this APIs. Then let's say you want to visit old chats, you are now adding a database for presistence. It opens a can of worms which are quite painful to deal with and will almost always give you a sub-par experience to using an LLM client like ChatGPT or Claude. Through MCP, adding new tools is very easy since you don't have to worry about interacting with the LLM APIs at all. All you need to do is change the tool config on your client and you can start using them. --- Quite a few companies have chat interfaces for their support channels, where a bot is triaging the issue, and if resolution is not possible, a human can intervene. But not many have adopted providing product features through chat, largely because having the situational awareness was not possible until LLMs. Through MCPs, it is now possible to have these features accessible through an LLM client interface. And no, you do not need to develop your own proprietary client that will call an LLM API. Rather, the LLM client will call your APIs. ### Product-led MCPs One of the first MCP servers was published by Cloudflare. Massive kudos to this team for their velocity in releasing features. Their MCP clients let you tweak Cloudflare resources from any MCP-supported client. The setup of their server is seamless. They authenticate your account during setup, and you can query your resources on Cloudflare. Without needing to open Cloudflare's dashboard, you can get values in their object storage, operate your crons, and even update route maps. Similarly, Stripe released their MCP, which had capabilities to create invoices, query information about customers, and so on. If not for MCP, building these experiences would have had to rely on creating custom LLM clients and providing them. Through MCP, they can now distribute their MCP to any supported client. Since these products don't explicitly need to build a client, they can engineer the vibe of their tools and resources to reflect their brand. The criteria for the tool invocation, and how the errors are handled can be crafted better so that while using through an LLM, the brand is not diluted to just the system prompt of the LLM client. ## 3. Runtime discovery of new capabilities Without MCP, providing tools to the LLM needs to be done at design time. When instantiating the LLM client, the tool definitions are provided. Through MCP, they can be discovered during runtime, and new tools can be added as required. ### MCP server discovery Quite a few companies offering hosted STDIO MCPs as well as HTTP MCPs have come up and are quite easy to integrate. They also keep an index of all MCPs available in the wild. Here are some of the popular websites: * [Mcp.run](https://mcp.run) * [Compose.io](https://compose.io) * [Smithery](https://smithery.ai) In a recent talk by Mahesh Murang from Anthropic, an official MCP server registry was announced that can be used for discovery and adding new tools during runtime. MCP server discovery interface showing available tools and capabilities ## 4. The learning curve for using and creating MCPs is minimal Support for MCP SDKs started off with Python and JS, and it has grown since to now support Java, C#, Go, Kotlin, and Rust (in development). The type definitions in MCP are straightforward to integrate and build your own MCP server. Testing is quite easy too. Through Claude desktop, you could fire a query about your tool, and it's invoked. Debugging through the MCP Inspector is easy. All the features in the specification are supported on the inspector. ### "MCP is too complex" For HTTP-based implementations, the developer feedback has been that it's too complex. Based on the first draft of the protocol, the SDKs were not as feature-complete for the HTTP implementations. There were quite a few issues on GitHub about people being unable to make SSE work easily. At launch, Inspector was the only MCP-supported client to test the SSE-based MCP servers. The MCP community has been responsive to this feedback, and the recent introduction of Streamable HTTP demonstrates this commitment to simplification while maintaining the protocol's core benefits. ### How Streamable HTTP Simplifies MCP The new Streamable HTTP transport introduced in the March 2025 specification revision makes the HTTP implementation completely stateless as compared to the long-standing connections that were required earlier. Now, it's a single consolidated endpoint with familiar POST and GET methods. And it's also fully backward compatible. With the improvement to HTTP transport on MCP, the adoption of remote MCP servers will improve. ## Possible risks The largest risk for MCP was that there would be competing standards and adoption would not take off. XKCD comic about competing standards With the recent vote of confidence from OpenAI to also adopt MCP, this seems like a great step in this direction. ### Authentication & Authorization The current authorization methodology in the specification expects the MCP server to maintain the client tokens. Essentially expecting a backend system to be stateful, this adds additional complexity. Having persistence of these tokens make the entire server extremely critical infra. Since it's early days, like the HTTP changes, this will also evolve. ## Closing thoughts If you're building LLM-powered applications and care about the user experience of structured outputs, check out my post on [making LLM workflows human friendly](/blog/streaming-function-calling/) — it covers how to validate streamed JSON in real-time so your users don't stare at a spinner. Play around with LLM clients that support MCP and see for yourself. It's fairly easy to set up. I've now come to extensively use the fetch tool quite often to provide context from a specific website to Claude. Start off with the `fetch` and `brave_search` tool, it's great addition to Claude for it to look up links, coupled with brave search, you can have a rudimentary deep-search. If you fancy further, try the playwright mcp by microsoft, to use a headless browser, you can even provide screenshots to claude and control the browser through the accessibility tree. ```json { "mcpServers": { "brave-search": { "command": "npx", "args": [ "-y", "@modelcontextprotocol/server-brave-search" ], "env": { "BRAVE_API_KEY": "" } }, "fetch": { "command": "uvx", "args": [ "mcp-server-fetch" ] } } } ``` --- # Making LLM workflows human friendly > Validate the data as it streams, don't make your users wait. Published: 2025-01-28 Author: Nikhil Kasukurthi Tags: LLM, Streaming, JSON, UX, Performance Canonical: https://bluenotebook.io/blog/streaming-function-calling/ --- ## LLMs = Intelligence + Latency? This lovely tweet by @vboykis is a great reminder of how LLMs are being perceived. GitHub copilot was one of the first to use the ✨ emoji to indicate that the response is being generated by an LLM. Now ✨ emoji and purple-hued gradients on icons and websites have become synonymous with intelligence, but also **latency** (engineering jargon for calling something slow). ## But, why? Why care about (LLM) latency? It is well established in Human Computer Interface (HCI) research that, after any user action, if the output or change takes longer than one second to show, it is perceived as being slow [+density]. [+density]: Ström, Matthew. 2024. "Density in Time." https://matthewstrom.com/writing/ui-density/ - discusses how users perceive time density in user interfaces. If the information is shown within 100ms, the time between action and outcome feels instant, for longer interactions within a second, having an animation or loading states help in reducing the user anxiety (or rather the impatience). Any interaction that takes longer than 10s makes the user lose interest. In the early days of ChatGPT, it was common for users to wait for 4-5s to even start seeing any response from the model. But since day 1 of ChatGPT, they have been showing the decoded output of the model as it's available, essentially stream native. In the model context protocol (MCP) implementation on Claude desktop, the text from tool-use is streamed well. Now with the growth of applications offering LLM intelligence, some of them feel quite sluggish because they are slow. But in reality, if we breakdown the network timing of an LLM call, you will see a majority of the time is being taken to decode the output tokens (content download). For the query - *Give me a recipe for light roast v60 pour over*, Claude takes 5s to complete the response but only 630ms for the first word (token) to be decoded.
Query with 20 tokens showing timing breakdown
(a) Query - 20 tokens
Timing breakdown for 360 output tokens showing first token vs total completion time
(b) Timing breakdown for 360 output tokens
In case of chat applications, streaming the text is straightforward, user asks a questions and the LLM generates free-flowing text without any pre-defined structure. However, in case of workflow driven cases where LLMs are generating some structured output, having the user wait for multiple seconds without any intermediate output is painful. This reminds of the XKCD comic about compiling.
XKCD comic about compiling - the modern day variant is LLM is decoding!
Instead of compiling, the modern day variant is **LLM is decoding!**
Keeping this in mind, let's first establish some metrics that are good to keep track while offering LLM based applications. ### Metrics to track for LLMs 1. Time to decode the first token 2. Total time for completion of text stream 3. Total count of output tokens (they are more expensive than input tokens) ## Structured output generation from LLMs While generating structured output from LLMs (tool-use, JSON mode, function calling), the common UX paradigm is to wait for the entire output to be generated before showing it to the user. Since validation of the output is required to ensure it's in the same schema as expected. Having the ability to validate the decoded text as it's being streamed can significantly elevate the entire experience, cutting down the waiting time. Most popular LLM providers (OpenAI, Anthropic, AWS Bedrock) and open source LLM serving implementations (like vLLM, Triton, MLXServing) provide streaming tokens (text) as it's generated by the model [+streaming] [+streaming]: Willison, Simon. 2024. "How streaming LLM APIs work." https://til.simonwillison.net/llms/streaming-llm-apis - explains the technical implementation of streaming in LLM APIs using Server-Sent Events. enabled through server sent events [+see]. [+sse]: Server-Sent Events. 2024. MDN Web Docs. https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events - the web standard that enables real-time streaming from server to client. --- Let's say we need to build an application that can digitize this image of a handwritten family recipe for rasam, into a structured format so that we can catalogue and show it on the UI shown below.
Traditional family recipe for rasam showing handwritten ingredients and instructions
Family recipe
Sample UI showing streaming recipe generation interface
Sample UI
### Why LLMs for this application? We are going beyond just Optical Character Recognition (OCR) and transforming basis the information architecture defined on the UI. The following things can be implicitly handled by an LLM: * Need to parse handwritten text * The ingredients are scattered throughout the image, so we need to capture all of them Now based on the sample UI above, we have a datastore of recipes that has the following schema: ``` Recipe: - title: String - ingredients: List of Ingredient - instructions: List of String Ingredient: - item: String - quantity: Integer|String - unit: Enum (kg, g, l, ml, tsp, tbsp, cup, piece) ``` ## Visualising streamed events from LLMs While the model decodes tokens as per the schema, seeing the updates visually will give a better sense in understanding the importance of validating the structured output as it's streaming. In the interactive widget below, click on `start streaming` button to start the decoding process and you can scrub through the timeline to see the changes in the data and how it looks on the UI. ### Things to observe * It takes about 4.5s to decode the entire schema but the important nuance is that **it only takes 100ms to decode the first token** [+firsttoken]. * Based on the pseudo-UI above, we are decoding the JSON to show the UI elements in that order: title, ingredients, and finally, recipe. [+firsttoken]: The time to first token (TTFT) is critical for user experience - it determines when users see the first sign of progress, significantly affecting perceived responsiveness. import StreamingTimelineIsland from '../../components/StreamingTimelineIsland.astro'; If you run the image above through sonnet-3.5-v1 using the Anthropic SDK, It's about 3000 input tokens and output would be about 300 tokens which is expected to be a JSON. ## How to: Structured output generation Now that we have established the way in which the data needs to be decoded, we need a way provide the expected schema to the LLM. Tool use or JSON mode is a reliable way to generate this structured output. You can also achieve the same without using function calling by just mentioning in the prompt to fill the schema provided, with any instruction following model, it will save you some tokens when you don't explicitly use function calling. But this tends to fail with smaller open-source models (Llama-3.1-7B) when the schema is complex with nested objects and arrays. All developer SDKs (OpenAI, Anthropic, Bedrock, Gemini) expect the input for the schema for a function to be provided in OpenAPI JSON spec [+openapi], one of the standard ways to represent function schemas as JSON. Once the schema starting getting longer and has nested structures, writing it directly in OpenAPI spec becomes tiring and is quite error prone. [+openapi]: OpenAPI Specification. 2024. Swagger. https://swagger.io/specification/ - the industry standard for describing REST APIs and function schemas. Pydantic has a great implementation for creating these tool schemas through their BaseModels [+pydantic]. All you need to do is call the `BaseModel.schema_json()` to get the OpenAPI compatible spec JSON through this. Later on in the post, we will also use the partial json validation that Pydantic offers. [+pydantic]: Pydantic BaseModel. 2024. https://docs.pydantic.dev/latest/concepts/models/ - provides data validation and settings management using Python type annotations. ```python # data_models.py from pydantic import BaseModel, Field from enum import Enum from typing import List, Optional class UnitEnum(str, Enum): kg = "kg" g = "g" l = "l" ml = "ml" tsp = "tsp" tbsp = "tbsp" cup = "cup" piece = "piece" class Ingredients(BaseModel): item: Optional[str] = None quantity: Optional[int|str] = None unit: Optional[UnitEnum] = None class Recipe(BaseModel): title: Optional[str] = Field(description="Title of the recipe", default=None) ingredients: Optional[List[Ingredients]] = Field(description="List of ingredients", default=None) instructions: Optional[List[str]] = Field(description="Instructions to make the recipe", default=None) ``` > **Note:** Note the use of `Optional` in all the fields. While partial validation of JSON, if any of *required* fields are missing then Pydantic raises an exception. This is a bit of a hack to get around it, but also note that this increases the number of tokens that are sent to the model. ## Partial JSON validation Pydantic has released support for validation of partial json, it's based on their pydantic-core library and uses the jitter library written in rust for actually parsing the JSON. We are using the `from_json` method defined in pydantic_core to perform our validation. ```python # partial_json_validation.py from pydantic_core import from_json from data_models import Recipe def validate_partial_json(streamed_text: str): if streamed_text != "": validated_dict = from_json(streamed_text, allow_partial=True) recipe = Recipe.model_validate(validated_dict) return recipe return None ``` Decoding json partially involves keeping track of the opening and closing of quotes, different types of brackets for objects and arrays in a stack and then checking which of those match. I tried replicating this in JS but it there are a lot of cases that need to be handled. The jitter library is quite well written, and code base is fun to read through. Now, to perform this validation on the stream, on every content-block streamed from the LLM API, call the `validate_partial_json` function. ```python # main.py from pydantic_core import from_json def validate_partial_json(streamed_text: str): if streamed_text != "": validated_dict = from_json(streamed_text, allow_partial=True) recipe = Recipe.model_validate(validated_dict) return recipe return None def validate_stream(time_between_checks = 0.003): streamed_text = "" start_time = time.time() last_check = time.time() # This is the stream from anthropic for response in stream._raw_stream: if response.type == "content_block_start": continue if response.type == "content_block_delta": streamed_text += response.delta.partial_json elapsed_time = time.time() - start_time last_check_time = time.time() - last_check # check if it's been time between checks since last check if last_check_time >= time_between_checks: last_check = time.time() validated_json = validate_partial_json(streamed_text) if validated_json is not None: yield validated_json print(elapsed_time) validated_json = validate_partial_json(streamed_text) yield validated_json ``` In the code, there is a parameter `time_between_checks` to check how frequently to perform the partial JSON validation, this is to ensure we are not blocking the CPU constantly by doing the validation and having enough decoded tokens. ## Structured schema generation checklist LLMs always generate the next token, so, if a JSON object key is being currently decoded, we know that the value of that key will be decoded next. 1. Structure your schema in a way that first UI component to render is being decoded first, like in the recipe example, we decode the title. 2. Arrays and nested objects are decoded as is. --- If you're interested in how LLMs gain new capabilities beyond text generation, read about [why Model Context Protocol (MCP) matters](/blog/why-mcp/) — it's the emerging standard for extending LLM clients with tools, resources, and runtime discovery.