Decision models (Jev)¶
The earlier chapters all use LLMs: we write a prompt, and the model writes text back. A decision model works differently. It takes a text and a typed question with a fixed set of answers, and it returns one answer together with a probability for each possible answer. It generates no text, so there is no prompt to write, no explanation to read, and no JSON to parse.
This chapter uses Jev, a hosted decision model from TypeSafe. Four things differ from the earlier chapters:
- The request holds a
state(the text) andquestionsinstead of a prompt. - The answer is always one of the options you listed.
- You pay for input tokens only. Jev costs $0.042 per 1M input tokens, and output tokens are free. For comparison, `gpt-5.6-luna` costs $0.20 per 1M input tokens and $1.20 per 1M output tokens.
- Jev has its own API and its own Python package. The
openaipackage cannot call it, so this is the one chapter where the earlier code does not carry over.
We will make a first call, go through the three question types, check the confidence value against hand labels, build a cascade that sends the uncertain items to an LLM, and compare the cases where Jev is a good fit with the cases where you should use an LLM instead. The structured output chapter solves the same problem (output that a program can read) with an LLM.
The outputs on this page come from a run on 2026-10-01 with the model jev-1.13.0 and version 0.7.2 of the typesafe-sdk package.
Your answers will differ slightly from the ones shown here, and from one run to the next, because the API has no seed parameter.
Setup¶
Install the dependencies and configure your API keys. This chapter uses two keys:
TYPESAFE_API_KEYfor Jev. You can create an API key in the TypeSafe console. As of October 2026, a new account comes with a $5 free credit and needs no payment method.OPENAI_API_KEYfor the LLM in the section "Cascade with an LLM". The other sections do not need it. If you only want to run those, remove this name from the list in the second cell below.
One full run of this chapter costs about $0.001 for the Jev calls and about $0.003 for the OpenAI calls.
On Google Colab, store your keys in Colab Secrets first: click the key icon in the left sidebar, add two secrets named TYPESAFE_API_KEY and OPENAI_API_KEY, and enable notebook access for both. Running locally, put the keys in a .env file or in environment variables with the same names (see handling API keys).
A note on research data. Jev is a hosted service, so every text you label is sent to TypeSafe. TypeSafe states that it does not train on customer input. It states no retention period, and it processes data in the United States. Check your data agreement or ethics approval before you send human-subjects text.
%pip install -q typesafe-sdk openai pydantic python-dotenv
import os
from dotenv import load_dotenv
load_dotenv() # loads a .env file from the working directory, if there is one
for key_name in ["TYPESAFE_API_KEY", "OPENAI_API_KEY"]:
if key_name not in os.environ:
try:
# On Google Colab, read the key from Colab Secrets
from google.colab import userdata
os.environ[key_name] = userdata.get(key_name)
except ImportError:
raise RuntimeError(f"Set {key_name} in your environment or in a .env file")
Basic query¶
Let's label the sentiment of the text message from the basics chapter. Compared with that chapter, there are three differences to notice:
- There is no prompt. The text goes into
state, andquestionsis a dictionary that maps a name you choose to a question. - Our question is a
Choice. It has a one-lineinstructionstext andcriteria, a dictionary that maps each option to its definition. The model must pick one of these options. - The answer has no explanation. It holds the chosen option, a probability for each option, and a
confidencevalue.
from typesafe_sdk import Choice, TypeSafeClient
text_message = "The service here is very good!"
criteria = {
"positive": "The message gives a favorable evaluation of the service overall",
"negative": "The message gives an unfavorable evaluation of the service overall",
"neutral": "The message gives no evaluation of the service, or the favorable and unfavorable parts have about equal weight",
}
sentiment_question = Choice(
instructions="The overall sentiment of the text message about the service",
criteria=criteria,
)
client = TypeSafeClient(model="jev-1.13.0") # reads TYPESAFE_API_KEY
response = client.system_one(
state=text_message,
questions={"sentiment": sentiment_question},
)
answer = response.answers["sentiment"]
print(f"Label: {answer.choice}")
print(f"Probabilities: {answer.probabilities}")
print(f"Confidence: {answer.confidence}")
print(f"Model: {response.model}")
print(f"Input tokens: {response.usage.input_tokens}")
print(f"Cost: ${response.usage.input_tokens * 0.042 / 1_000_000:.6f}")
Label: positive
Probabilities: {'neutral': 0.0, 'negative': 0.0, 'positive': 1.0}
Confidence: 1.0
Model: jev-1.13.0
Input tokens: 365
Cost: $0.000015
We use jev-1.13.0 in this chapter.
It costs $0.042 per 1M input tokens, and output tokens are free.
The response has no cost field, so we compute the cost from the input tokens.
The alias jev-latest always points to the newest version.
We pin the version because the confidence threshold below was chosen for this version and may not hold for the next one.
The model field of the response tells you which version answered.
Using OpenRouter instead. If you already have an OpenRouter key from the open-source models chapter, you can call Jev with it. Only the client changes:
client = TypeSafeClient(
api_key=os.environ["OPENROUTER_API_KEY"],
base_url="https://openrouter.ai/api",
model="typesafe/jev-1.13",
)
The questions and the answer fields stay the same. The model id is OpenRouter's name for the same model.
Question types¶
Jev has three question types.
Each one is a class in the typesafe_sdk package.
| Type | Use it for | criteria |
Answer |
|---|---|---|---|
Noul |
A yes/no question | Optional | noul: the probability of yes, from 0 to 1 |
Choice |
Picking one option | A dictionary that maps each option to its definition | choice: the most probable option, with probabilities and confidence |
Score |
Rating on an ordered scale | A list of 2 to 10 level descriptions, from the lowest level to the highest | score: a number between 0 and the index of the highest level, with probabilities and confidence |
One request can hold many questions, and the model answers each of them independently. Let's ask three questions about our text message in one request: whether it is a complaint, its sentiment, and how positive it is on a scale with five levels.
from typesafe_sdk import Noul, Score
questions = {
"is_complaint": Noul(instructions="The message is a complaint"),
"sentiment": sentiment_question,
"positivity": Score(
instructions="How positive the message is",
criteria=[
"Very negative",
"Somewhat negative",
"Neutral or mixed",
"Somewhat positive",
"Very positive",
],
),
}
response = client.system_one(state=text_message, questions=questions)
answers = response.answers
print(f"Complaint: {answers['is_complaint'].noul}")
print(f"Sentiment: {answers['sentiment'].choice}")
print(f"Positivity: {answers['positivity'].score}")
print(f"Positivity probabilities: {answers['positivity'].probabilities}")
print(f"Input tokens: {response.usage.input_tokens}")
Complaint: 0.01
Sentiment: positive
Positivity: 3.98
Positivity probabilities: {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.02, 4: 0.98}
Input tokens: 435
The noul value is the probability that the answer is yes, so a value near 0 means that the message is not a complaint.
The levels of a Score question are numbered from 0, so our scale runs from 0 (very negative) to 4 (very positive).
The score is the average of these level numbers, weighted by their probabilities, so it can fall between two levels.
The request with three questions used 435 input tokens, and the request with one question used 365. Adding a question to a request costs much less than sending a second request.
Now let's run the same three questions on five text messages, from the most positive to the most negative.
text_messages = [
"The service here is very good!",
"The service here is good.",
"The service here is ok.",
"The service here is not very good.",
"The service here is terrible!",
]
for text_message in text_messages:
response = client.system_one(state=text_message, questions=questions)
answers = response.answers
print(
f"{text_message:<36}"
f"complaint: {answers['is_complaint'].noul:.2f} "
f"sentiment: {answers['sentiment'].choice:<10}"
f"positivity: {answers['positivity'].score:.2f}"
)
The service here is very good! complaint: 0.01 sentiment: positive positivity: 3.98 The service here is good. complaint: 0.02 sentiment: positive positivity: 3.03 The service here is ok. complaint: 0.14 sentiment: neutral positivity: 2.09 The service here is not very good. complaint: 0.98 sentiment: negative positivity: 0.92 The service here is terrible! complaint: 0.99 sentiment: negative positivity: 0.00
The three answers change together as the messages become more negative.
Two things to keep in mind when you write your own questions:
- A
Choicealways returns one of the listed options. If your options may not cover every text, add an option such asother. - There is no question type for several labels at once. For a multi-label task, ask one
Noulquestion per label in the same request.
Confidence¶
Every Choice answer has a confidence value from 0 to 1, which is computed from the probabilities.
A value of 1 means that all the probability is on one option.
A value of 0 means that the options are equally probable.
The value is useful only if the labels with high confidence are correct more often than the others. The way to find out is to compare Jev's labels with labels that we trust.
The file text_message_labels.jsonl holds 60 short text messages about services.
Each one has a label: positive, negative, or neutral.
We drafted the messages and their labels with an AI assistant for this chapter, and the author reviewed every label by hand.
We call these the hand labels.
They follow the three definitions in our criteria, and they describe the evaluation of the service only.
A message that evaluates only something else, such as the weather, is neutral.
This is why our question names the service.
Of the 60 items, 36 are clear and 24 are ambiguous on purpose: mixed evaluations, sarcasm, negation, hedged wording, questions, and evaluations of something other than the service.
The cell below reads the file from the current folder if it is there. Otherwise it downloads the file from GitHub, which is what happens on Colab.
import json
import urllib.request
data_file = "text_message_labels.jsonl"
data_url = "https://raw.githubusercontent.com/yang3kc/llm_for_css/main/docs/text_message_labels.jsonl"
if os.path.exists(data_file):
with open(data_file, encoding="utf-8") as f:
lines = f.read().splitlines()
else:
with urllib.request.urlopen(data_url) as f:
lines = f.read().decode("utf-8").splitlines()
items = [json.loads(line) for line in lines]
print(f"{len(items)} items")
for item in items[:5]:
print(f"{item['label']:<10}{item['text']}")
60 items positive The service here is very good! neutral We arrived at 7 and left at 9. negative Oh great, another 40-minute wait for a cold coffee. negative My order arrived two days late and the box was crushed. neutral The food was great, but we waited an hour for a table.
Now we label every item with the sentiment question from above.
A plain loop is enough for 60 items.
For each item we keep the label, the confidence, and the number of input tokens.
jev_results = []
for item in items:
response = client.system_one(
state=item["text"],
questions={"sentiment": sentiment_question},
)
answer = response.answers["sentiment"]
jev_results.append(
{
"label": answer.choice,
"confidence": answer.confidence,
"input_tokens": response.usage.input_tokens,
}
)
n_correct = sum(result["label"] == item["label"] for item, result in zip(items, jev_results))
print(f"Jev gives the hand label on {n_correct} of {len(items)} items.")
Jev gives the hand label on 59 of 60 items.
Next we split the items at a confidence of 0.9 and compute the accuracy of each group. We fixed this limit before we ran the model, and we use it again as the threshold of the cascade in the next section. With only 60 items, two groups are as much as the data can support. With more items, you can use more groups.
The cell also lists every wrong label, with the highest confidence first.
threshold = 0.9
pairs = list(zip(items, jev_results))
bands = {
f"{threshold} and above": [pair for pair in pairs if pair[1]["confidence"] >= threshold],
f"below {threshold}": [pair for pair in pairs if pair[1]["confidence"] < threshold],
"all items": pairs,
}
print(f"{'Confidence':<16}{'Items':>6}{'Share':>8}{'Accuracy':>10}")
for name, band in bands.items():
n_correct = sum(result["label"] == item["label"] for item, result in band)
accuracy = f"{n_correct / len(band):.0%}" if band else "n/a"
print(f"{name:<16}{len(band):>6}{len(band) / len(pairs):>8.0%}{accuracy:>10}")
errors = [pair for pair in pairs if pair[1]["label"] != pair[0]["label"]]
n_confident = sum(result["confidence"] >= threshold for item, result in errors)
print(f"\nWrong labels: {len(errors)}, of which {n_confident} with a confidence of {threshold} or above")
for item, result in sorted(errors, key=lambda pair: -pair[1]["confidence"]):
print(
f"{result['confidence']:.2f} hand label: {item['label']:<10}"
f"Jev: {result['label']:<10}{item['text']}"
)
Confidence Items Share Accuracy 0.9 and above 51 85% 100% below 0.9 9 15% 89% all items 60 100% 98% Wrong labels: 1, of which 0 with a confidence of 0.9 or above 0.80 hand label: neutral Jev: positive I love my new phone. I used it to book a table with you for Friday.
How to read this output:
- Accuracy is higher in the top group. Jev gives the hand label on all 51 items with a confidence of 0.9 or above, and on 8 of the 9 items below. So the confidence value can decide which Jev labels to keep, and the share column shows that this covers 85% of the items.
- A high confidence does not mean that the label is correct. In this run no wrong label has a confidence of 0.9 or above, but that is a result on 60 short messages, and it is not a guarantee. On an empathy task, Ibrahim and Zaki (2026) found that 78% of the items had a confidence of 0.9 or above, and the accuracy on those items was 0.383, against a base rate of 0.371.
- The limit and the result hold for this task and this model version only.
This check is a demonstration. The 60 items were written for this chapter, and they do not measure how accurate Jev is on real data. For your own task, label a small sample by hand and repeat the check before you rely on a threshold. The three preprints listed at the end of this chapter study the calibration of Jev in depth.
Yes/no questions. A Noul answer has no confidence field. If you need a confidence value for a yes/no question, ask it as a Choice with the two options yes and no. The answer then has a confidence field, and the check above and the cascade below work in the same way. The two forms of one question can give different probabilities: the vendor documents a question that gave 0.22 as a Noul and 0.01 for yes as a Choice. A threshold that you find for one form therefore does not carry over to the other.
Cascade with an LLM¶
A cascade combines a cheap model with a more expensive one:
- Jev labels every item. We already have these labels.
- The items with a confidence at or above a threshold keep the Jev label.
- The other items go to an LLM.
The LLM is gpt-5.6-luna, and the call is the one from the structured output chapter.
The Pydantic model has one field, label, and Literal limits it to our three labels.
We build the label definitions in the prompt from the same criteria dictionary as the Jev question, so both models read the same definitions.
In a real cascade you call the LLM only for the items below the threshold. Here we send all 60 items to the LLM, so that we can also report how the LLM does alone.
from typing import Literal
from openai import OpenAI
from pydantic import BaseModel
class SentimentLabel(BaseModel):
label: Literal["positive", "negative", "neutral"]
system_prompt = "You are an expert on sentiment analysis. Your job is to label the sentiment of the given text message."
label_definitions = "\n".join(f"- {label}: {definition}" for label, definition in criteria.items())
openai_client = OpenAI()
llm_results = []
for item in items:
user_instruction = f"""
Given the following text message: '{item["text"]}', please choose the label that describes its overall sentiment about the service.
The labels are:
{label_definitions}
"""
response = openai_client.responses.parse(
model="gpt-5.6-luna",
instructions=system_prompt,
input=user_instruction,
text_format=SentimentLabel,
)
llm_results.append(
{
"label": response.output_parsed.label,
"input_tokens": response.usage.input_tokens,
"output_tokens": response.usage.output_tokens,
}
)
n_correct = sum(result["label"] == item["label"] for item, result in zip(items, llm_results))
print(f"The LLM gives the hand label on {n_correct} of {len(items)} items.")
The LLM gives the hand label on 57 of 60 items.
Now we build the cascade labels. The threshold is 0.9, the limit from the confidence table. An item keeps its Jev label when the confidence is at or above the threshold, and takes the LLM label otherwise.
cascade_labels = []
sent_to_llm = []
for item, jev, llm in zip(items, jev_results, llm_results):
if jev["confidence"] >= threshold:
cascade_labels.append(jev["label"])
else:
cascade_labels.append(llm["label"])
sent_to_llm.append((item, jev, llm))
n_llm_calls = len(sent_to_llm)
print(f"{n_llm_calls} of {len(items)} items go to the LLM.")
print(f"Jev gives the hand label on {sum(jev['label'] == item['label'] for item, jev, llm in sent_to_llm)} of them.")
print(f"The LLM gives the hand label on {sum(llm['label'] == item['label'] for item, jev, llm in sent_to_llm)} of them.")
9 of 60 items go to the LLM. Jev gives the hand label on 8 of them. The LLM gives the hand label on 6 of them.
Finally we compare Jev alone, the LLM alone, and the cascade. For each one we report the accuracy against the hand labels, the number of LLM calls, and the cost per 1,000 items. The cost comes from the token counts in the responses and the list prices. The cascade pays Jev for every item and the LLM for the items below the threshold.
jev_price = 0.042 / 1_000_000 # US dollars per input token
llm_input_price = 0.20 / 1_000_000
llm_output_price = 1.20 / 1_000_000
jev_cost = sum(result["input_tokens"] for result in jev_results) * jev_price
llm_costs = [
result["input_tokens"] * llm_input_price + result["output_tokens"] * llm_output_price
for result in llm_results
]
llm_cost = sum(llm_costs)
cascade_cost = jev_cost + sum(
cost for cost, jev in zip(llm_costs, jev_results) if jev["confidence"] < threshold
)
rows = [
("Jev alone", [result["label"] for result in jev_results], 0, jev_cost),
("LLM alone", [result["label"] for result in llm_results], len(items), llm_cost),
("Cascade", cascade_labels, n_llm_calls, cascade_cost),
]
print(f"{'':<12}{'Accuracy':>10}{'LLM calls':>12}{'Cost per 1,000 items':>24}")
for name, labels, n_calls, cost in rows:
accuracy = sum(label == item["label"] for label, item in zip(labels, items)) / len(items)
cost_per_1000 = f"${cost / len(items) * 1000:.3f}"
print(f"{name:<12}{accuracy:>10.1%}{n_calls:>12}{cost_per_1000:>24}")
print(f"\nCost of this run for {len(items)} items: ${jev_cost:.4f} for Jev and ${llm_cost:.4f} for the LLM")
Accuracy LLM calls Cost per 1,000 items Jev alone 98.3% 0 $0.016 LLM alone 95.0% 60 $0.051 Cascade 95.0% 9 $0.024 Cost of this run for 60 items: $0.0009 for Jev and $0.0031 for the LLM
On these 60 items the cascade does not help. Jev alone gives the hand label on 59 of the 60 items (98.3%), and it is also the cheapest option. The LLM alone reaches 95.0% and costs about three times as much ($0.051 against $0.016 per 1,000 items). The cascade sends 9 items to the LLM. Jev gives the hand label on 8 of them and the LLM on 6, so the cascade replaces correct Jev labels with wrong LLM labels and ends at 95.0%. All 3 wrong LLM labels are among these 9 items: the messages that Jev is unsure about are also the ones that the LLM gets wrong.
This is one result on a small set, and gpt-5.6-luna is a small model.
A cascade pays off only when the LLM is more accurate than Jev on the items below the threshold, and the second cell above shows how to check that.
With a stronger LLM the result can be different: on judging tasks, Li et al. (2026) report a cascade from Jev to GPT-6 that kept about 99% of the accuracy of GPT-6 at about 57% of its cost.
Two more things limit what this table tells you:
- We kept the threshold after we saw the confidence table for the same 60 items. A threshold that is chosen and tested on the same items gives a result that is too optimistic. In a project, choose the threshold on one labeled sample and report the result on another.
- A cascade gains little when the LLM makes the same errors as Jev. Rao and Callison-Burch (2026) found this for rubric grading, where the LLM judges repeated most of Jev's confident errors.
Jev or an LLM?¶
Jev and an LLM suit different tasks.
Everything in this section describes jev-1.13 only.
The three preprints are from September 2026, and the vendor's limitations page was last reviewed on 2026-09-17.
A later version of Jev may behave differently.
Jev is a good fit when:
- The task is to label text with a fixed set of answers, and you have many items. This is the task that a decision model is built for. You write the options and their definitions once, and the same question runs on every item.
- Cost and time matter. In a study that compared Jev with 19 LLMs on 15 social science annotation tasks, Jev was a median of 44 times cheaper (Ibrahim and Zaki, 2026). In rubric grading, three low-cost LLM judges cost 29 to 325 times as much as Jev and took 30 to 220 times as long, and the accuracy of Jev was not significantly different from theirs in 19 of 27 comparisons (Rao and Callison-Burch, 2026).
- You want a probability for each label. Every answer comes with probabilities, and a
ChoiceorScoreanswer also has a confidence value. You can keep the confident labels and check the others yourself or send them to an LLM. You must still compare the confidence with hand labels first, as in the Confidence section. - You want output that needs no parsing or validation. The answer is always one of the options you listed. In the annotation study, the decision models returned no invalid answer in 23,703 calls, and an LLM with a JSON schema still returned 16 invalid answers on the same tasks (Ibrahim and Zaki, 2026).
Use an LLM when:
- You need text back. Jev generates no text. It cannot give you an explanation or a summary, and it has no question type for extracting text (limitations page).
- The question needs counting, numbers, or a comparison of dates and times. The vendor lists all three as unreliable (limitations page).
- The question needs several steps, is indirect, or holds a double negative. The vendor lists these as failures. On the reasoning part of the JudgeBench benchmark, Jev reached 68.4% accuracy and GPT-6 reached 95.9% (Li et al., 2026).
- The construct is affective, such as empathy. On an empathy task, 78% of the items had a confidence of 0.9 or above, and the accuracy on those items was 0.383, against a base rate of 0.371 (Ibrahim and Zaki, 2026).
- Accuracy matters more than cost. On the 15 annotation tasks, Jev scored lower than the best of the 19 LLMs on 14 tasks, by a median of 11.6 macro-F1 points (Ibrahim and Zaki, 2026). The gap is smaller against small LLMs, as the rubric grading result above shows.
- The text is long and holds much irrelevant detail. The vendor reports that this lowers accuracy (limitations page).
- The text is not in English, or the input is not text. Jev takes text only. Its documentation says that English is the main training language and that other languages are handled "but not equally well".
A cascade is not always better than a single model either. When the LLM repeats Jev's errors, the cascade gains little: in the rubric grading study, no cascade beat the best single judge by more than 1.5 percentage points (Rao and Callison-Burch, 2026).
For a new task, label a small sample by hand and run the check from the Confidence section before you choose a model.
References
- Ibrahim, H., & Zaki, Y. (2026). Evaluating Decision Models for Text Annotation in Computational Social Science. arXiv:2609.24574.
- Li, Y., Miao, Y., Krishnan, R., & Padman, R. (2026). JEV-as-a-Judge: Accept When Confident, Escalate When Unsure. arXiv:2609.26550.
- Rao, D., & Callison-Burch, C. (2026). JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places. arXiv:2609.29769.