Use case

Jev document classification

Most classification work is not hard, it is voluminous. The question is rarely whether a model can label a page, but what it costs to label a million of them.

The shape of a classification call

One call carries one unit of content and a closed label set. The unit might be a page, a row, a message, or a short document. The answer is a label plus a calibrated probability, which is what makes a threshold possible: the confident majority is decided, and whatever falls below the cutoff goes somewhere more expensive.

That two-part structure is what separates this from a chat completion. A chat model can classify, and often does it well. But it returns prose that you then have to parse, and you pay for the parse twice — once in tokens and once in the code that has to handle a response that ignored your format.

questions = {
  topic: { type: choice, criteria: { invoice, contract, receipt, other } },
}
// answer -> { choice, confidence, probabilities }
// then, in your code:
if (answer.confidence < 0.9) queueForLargeModel(item)

What the reported per-item numbers actually say

The figures in the case list below span more than an order of magnitude per item, from about $0.001 per page to roughly $0.05 for a couple of thousand short articles. None of them states the document length, the size of the label set, or whether the calls were warm, and those three variables move the cost more than the choice of model does.

Read them as existence proofs rather than a price list. What they establish is that per-item costs land where a pipeline can afford to classify everything instead of pre-filtering with heuristics — that is the decision these numbers should inform.

What we measured on the same task

We ran a six-way classification task over a fixed 16-item set against two chat models, and published every response. Accuracy was at ceiling for all three systems, 94% to 100%, so it told us nothing. Cost did: Jev came out at $0.0200 per thousand decisions, the cheap chat baseline at $0.0351, and the frontier baseline at $0.1497.

The honest reading is narrower than the circulating claims. Against the frontier model that is a 7x saving; against the cheap model it is under 2x. The 400x figures you see quoted are not reproducible on a short-input classification task, and our inputs were deliberately short — around 475 tokens. Per-document cost scales with length, and Jev charges about $0.042 per million input tokens with no charge for output, so its advantage should grow with document length. That is a prediction our current data does not test, and we would rather say so than imply it.

One number that does travel: Jev used roughly 2.3 times as many input tokens as the chat baselines for the same task, because a structured state plus a criteria map is more verbose than a sentence. If you are budgeting by tokens rather than by requests, account for that.

Batching without losing the tail

The pattern that shows up in the better builds is a two-stage pipeline: classify with a decision model, then route only the low-confidence remainder to something larger. One build classified a hundred messages, sent the uncertain ones to a frontier model, and reported 96 correct out of 100 for roughly seven cents.

The economics of that pipeline depend entirely on the size of the tail, which is why the confidence distribution deserves its own measurement before you commit to a threshold. In our task set, the share of Jev decisions scoring below 0.9 varied by task and was not uniform, so a single global cutoff would have been wrong for at least one of them. Measure the distribution per label set, not once for the whole system.

for (const batch of chunks(items, 500)) {
  const results = await Promise.all(batch.map((item) => jev({
    state: { text: item.text },
    questions: { topic: { type: 'choice', criteria: LABELS } },
  })))
  const confident = results.filter((r) => r.topic.confidence >= THRESHOLD)
  const tail      = results.filter((r) => r.topic.confidence <  THRESHOLD)
  write(confident); queueForLargeModel(tail)   // the tail is where the money goes
}

Failure modes, in the order they bite

Long documents break the economics before they break the model. A forty-page PDF sent whole is a large input on every call, and at that point you are paying for tokens you did not need — most of the informative content is in a fraction of the pages. Splitting into pages or sections and classifying the parts is cheaper and gives you a confidence score per part rather than per document.

Label sets drift. The moment someone adds a fifth category without re-measuring the confidence distribution, your threshold is calibrated to a different problem. Treat the label set as part of the configuration you version, and re-measure when it changes.

Content that is not really a decision will not become one. Pure extraction — get the invoice number off this page — is a different task with a different failure profile, and the reported accuracy on classification does not transfer to it.

Finally, the tail is not a rounding error. If 30% of your documents land below the threshold, your pipeline cost is dominated by the expensive model and the decision model is mostly acting as a filter. That can still be the right design, but it is a different design, and the share is the number that tells you which one you built.

Where this sits in the wider pattern

Document classification is the agent-loop pattern with the decision taken out of the loop and applied to a queue. If you are choosing between routes or tools rather than labels, start there. If the input is a single short message rather than a document, the inbox triage page is closer.

Reported on X

Quoted metrics are as posted by the author and are not verified or normalised by us.

CLI that sorts a folder by file content, with filenames stripped so only content could inform the label.
Reported: 2,225 articles in 4.9s, $0.05, 97% accuracy
@mikemenard_com on X
Two-stage pipeline: summarise with a chat model, then classify into 24 topics with Jev.
Reported: $0.08 total, 256ms median end-to-end per paper
@nutlope on X
Tax-document page classifier replacing an existing LLM pipeline.
Reported: $0.001 per page, 34× cheaper, 6× faster
@nedwize on X
Resume screening with typed scores and shortlist signals.
Reported: 360 resumes in 24.2s, $0.0212
@YouWareAI on X
Row classification exposed as a database extension, so analysts call it in SQL.
Reported: ~10s for 1k rows
@hamiltonulmer on X