Models Keep Getting Bigger. Why Do We Still Need GPU Partitioning?

Bigger models make GPU partitioning matter more.

• AI Engineering • 9 Minute • 2016 words • Recently Updated

Models getting bigger and GPU partitioning getting more important are two things that can be true at the same time.

A Question from Sheng Liang

This September, at the GPUStack ecosystem conference in Beijing, I found myself chatting with their CTO, Sheng Liang, over the conference dinner. He asked me one question: in the era of large models, is GPU virtualization and partitioning still that important?

He asked it politely, but for someone who works on the HAMi community and its commercialization, it stung. Everyone understands the subtext: models no longer fit on a single card, and you people are still figuring out how to slice one card into pieces. Isn’t that doing aerodynamics research on a wagon wheel?

I gave a polite non-answer that evening, but the question kept bugging me, and I realized it deserved a proper written response. As it happens, in mid-September TypeSafe AI released Jev, a model that cannot chat, and that made the question much more interesting. This post is my formal answer.

Jev: A Model That Cannot Chat

First, what Jev is. TypeSafe calls it a System One Model: you send an unstructured state plus a set of typed questions, and the output is not free text but three predefined types, Choice (pick one from a list), Score (rate on a scale), and Noul (yes or no), each with a probability distribution and confidence.

Why do I say it “cannot chat”? Consider how we use a large language model (LLM) today to decide “is this ticket about billing”. Asking a big model to do this is like hiring a Pulitzer winner to fill in a multiple-choice sheet: they will absolutely write you a heartfelt short essay first, and then you scrape the answer out of it with a regex. To get them to say “yes”, we first taught them to write everything, then strapped them to a chair with JSON mode, and then wrote a parser to guard against improvisation.

Jev deletes that entire pipeline. It does no autoregressive generation; it returns typed results that code can consume directly. Multiple questions can go into a single call, evaluated in parallel against the same state. The official docs claim that “adding questions barely changes the response time”, because each question is evaluated independently and cannot pollute the others’ context.

Figure 1: Generative LLM vs Jev output paths
Figure 1: Generative LLM vs Jev output paths

An analogy: a large LLM is like a senior consultant who reads materials and writes reports. Jev is like the real-time approval node inside a company. It will not write your report, but it can process huge volumes of “approve / reject”, “A/B/C”, “risk 0 to 10” judgments per second. You would not hire a novelist to run your access control system, yet that is exactly the architecture many companies run today.

Two buckets of cold water, as usual. First, “Jev cannot hallucinate” needs de-noising: type safety guarantees the output stays within the predefined schema, not that the business judgment is correct. A perfectly valid high_risk can still be a mistake. Second, the “193.6x faster, 444.6x cheaper” numbers on the homepage come from workflows designed by TypeSafe’s own team, and the company itself admits they sit at the high end of real-world gains. They indicate how much headroom the new paradigm has, not that “Jev is 200x faster than GPT”. One more easily misread number: 250,000 tokens/s is an API rate limit, not measured single-GPU throughput. API throughput, model throughput, and GPU throughput have never been the same thing.

Try It Yourself

Reading introductions gets you nowhere; run it to get the feel. console.typesafe.ai is open for registration. Create an API key, then:

pip install typesafe-sdk
export TYPESAFE_API_KEY=your_key

The example in the official docs is literally a support ticket classifier, three questions in one call:

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state={"document": "I was charged twice. Please fix this ASAP."},
        questions={
            "billing": Noul(instructions="Is this ticket about billing?"),
            "tone": Choice(
                instructions="What is the customer's tone?",
                criteria={"calm": None, "frustrated": None, "angry": None},
            ),
            "urgency": Score(
                instructions="How urgent is this ticket?",
                criteria=["can wait", "this week", "today"],
            ),
        },
    )

print(response.nouls["billing"].noul)   # yes/no decision
print(response.choices["tone"].choice)  # selected option
print(response.scores["urgency"].score) # rating level

Notice the shape of this API: no prompt, no temperature, no system message. You submit questions and options; you get back types and probabilities. If you have written LLM apps for a while, it feels like something is missing at first glance. Think again, and you realize what is missing was never supposed to be there.

The ecosystem is assembling faster than I expected. Beyond the official Python and TypeScript SDKs, the Rust community has typesafe_ai_rs, Pydantic’s docs already show a TypeSafeModel integration, and someone has built an MCP server for Jev. If you would rather not register directly, OpenRouter, Cloudflare Workers AI, and Vercel AI Gateway can all reach it. Chinese WeChat tech columns are filling up with Jev explainers too. The heat is real.

The Open-Source World Is Not Idle Either

More interesting: you can already reproduce this approach with open source. Bespoke Labs’ Nimble is one example, a LoRA adapter trained on Qwen3.5-9B, about 165 MiB, Apache 2.0. Its method is blunt: score the allowed answer tokens directly, and generate no free-form text at all during inference.

from inference import NimbleModel

model = NimbleModel("nimble-model")
result = model.score(
    context="The store accepts returns within 30 days. "
            "This item was bought 12 days ago.",
    schema={
        "eligible": {
            "type": "boolean",
            "description": "Is this item within the store return window?",
        }
    },
)
print(result["fields"]["eligible"]["probabilities"])

There are constraints: at most 26 choices per field, inputs over 2,048 tokens are rejected rather than truncated, and you need a CUDA GPU. But its existence proves a point: the so-called System One paradigm is not black magic. Take a general model, constrain its output space, score candidate answers directly, and you get something 70 or 80 percent of the way there for the cost of a LoRA. Which in turn makes Jev’s pricing worth studying: $0.042 per million input tokens, output free, with the homepage proudly noting it is 238x cheaper on input than a certain frontier model. Competitive pressure will likely arrive faster than expected.

Where Small Models Belong: Judgment, Not Chat

Now back to architecture. The real value of this class of models, I think, is not replacing a 70B model with a 3B one, but decomposing the intelligence workload that one big model used to monopolize. An enterprise agent may contain dozens of judgments: intent recognition, permission classification, document relevance, content risk, tool routing, whether to retry, whether to hand off to a human. These do not need frontier-level capability on every single call.

The more sensible division of labor: code owns deterministic logic, specialized decision models own high-frequency judgments, small models own local language tasks, and big models handle only genuinely hard reasoning. NVIDIA Research’s position paper on SLMs and agentic AI says the same thing: most calls inside an agent are repetitive and specialized, and a heterogeneous model system is more economical than “call the same big model for everything”.

Figure 2: Dividing labor between code, decision models, and LLMs
Figure 2: Dividing labor between code, decision models, and LLMs

Expand the earlier ticket example: code does authentication and field validation first, then asks a decision model three questions in one call. Is it billing? What tone? How urgent? With enough confidence, route and execute directly; refunds and database writes stay in code. Only low-confidence or open-ended cases escalate to a big model.

What I like about this architecture is precisely that it is not “more AI”. It hands system control back to software. Follow this direction and applications grow a distinct judgment plane: a layer of low-latency, confidence-carrying decision models sitting between code and big models. The API Gateway decides where traffic goes; the judgment plane decides who should think about this request. Jev is an early implementation of this direction, and traditional classifiers, rerankers, SLM routers, and Nimble above all belong to the same layer.

Back to Sheng’s Question

Now the direct answer: as models get bigger, does GPU partitioning still matter?

Once you accept that “one application runs a dozen models at once”, the GPU layer’s problem changes. A frontier model may need 8x H200, while the embedding model next to it needs a few GB of memory; rerankers, small vision models, guardrails, and notebooks each consume a sliver of GPU. If you still use Kubernetes’ traditional whole-card semantics:

resources:
  limits:
    nvidia.com/gpu: 1

then that is a commuter who calls a ride-share and gets a 49-seat bus every morning, with boarding forbidden for anyone else. Small models waste; many models fragment.

“Models are getting bigger, so GPU partitioning is obsolete” confuses two opposite directions on the resource axis.

Model parallelism solves one workload using many cards. Tensor Parallel, Pipeline Parallel, and Ray distributed inference all belong here: one card is not enough, use multiple cards on the same node; still not enough, go cross-node.

GPU partitioning solves many workloads sharing one card. HAMi (a CNCF Incubating project) turns the GPU from an integer device into a resource schedulable by memory, compute cores, and device share, pins the exact card at scheduling time, auto-matches MIG templates by compute and memory requirements, and enforces real resource constraints at the CUDA layer via HAMi-core rather than just numbers in the scheduler’s ledger. At the end of the day, split versus join is not just a question of direction. How finely you can split is the real engineering divide.

Figure 3: Two directions of the GPU resource axis
Figure 3: Two directions of the GPU resource axis

Here is the fun fact: the industry has already answered half of Sheng’s question for him. Platforms that aggregate models, the ones stitching big models across many cards, are shipping GPU partitioning and flexible slicing in the same product. Joining cards together with one hand while slicing them apart with the other is not a split personality; it is reality: demand exists in both directions at once, and picking a side is the mistake.

Looking further out, heterogeneity only gets worse. MoE decouples total parameters from active ones: Qwen3-30B-A3B has about 30B total parameters but activates only about 3B per token. 4-bit quantization and distillation keep pushing the real per-request cost down. Work like DistServe and Mooncake even splits prefill, decode, and KV cache across separate resource domains. Scheduling units will get finer, not coarser.

That said, do not treat GPU utilization as the only KPI. Cramming four inference instances into one card, taking nvidia-smi from 30% to 90%, does not automatically improve total cost of ownership; if P99 latency degrades and the failure domain widens, that utilization is worth little. HAMi fits GPU pools with obvious fragmentation, multiple tenants, and a high share of dev and test. For services with strict latency SLOs and cards saturated long-term, exclusive ownership or hardware isolation like MIG is the safer bet.

Conclusion

Jev’s significance is not that “small models won”, but that not every kind of intelligence needs to generate language. When one decision model or one LoRA adapter can handle half the checkbox work in an application, applications shift from “one big model does everything” to a dozen small models, each doing its job.

Big models keeping growing will not eliminate small models or GPU virtualization; heterogeneous model architectures mean the same cluster simultaneously faces “model too big” and “task too small” resource problems.

So the core capability of next-generation AI Infra is not merely slicing GPUs, nor merely stacking them, but freely splitting and composing compute resources when the workload demands it. In one sentence: split the small and fragmented, join the large and saturated, isolate the high-SLO, share the low-utilization.

So here is my answer: yes, and more than ever.

References

Jimmy Song

Jimmy Song

Focusing on research and open source practices in AI-Native Infrastructure and cloud native application architecture.

Updated on Sep 23, 2026 AI Infrastructure GPU HAMi Inference LLM

Post Navigation

Related Articles

Comments