Writing
jevdecision modeltrainingLLMs

jevlite - A decision model based on LLMs

Suppose you want software to route a support ticket. You have two teams (billing and technical), and a ticket:

Help! My payouts have been failing for 3 days.

So you ask an LLM which team should handle it, and you get back:

Based on the details provided, this appears to be a billing-related issue, though it could also involve technical factors. I’d recommend…

You send it a JSON structure, and it returns valid JSON about 97% of the time and you need to get to smarter and expensive models or they literally hallucinate your company structure for you and route it to a choice like customer_support which was not one of the valid options.

Jev is meant to solve that. TypeSafe AI’s Jev has taken the AI engineering world by storm. They have product and a service called Jev that is an order of a magnitude cheaper and faster for such structured decisions and it literally cannot hallucinate. I strongly encourage you to try it. They say imitation is the sincerest form of flattery. The TypeSafe team should consider themselves flattered.

Introducing jevlite

jevlite does the same thing that Jev does but is open source and you can run it today on your machine and poke at its internals. It is built on the gemma-4-e4b-it model and I trained a QLoRA adapter over 20k options for 2 hours to get similar results. This blog post is an explanation of how it works.

TypeSafe has not published the inner workings of Jev so I took my best guess. I could not have built this without the amazing work of the open source and open model ecosystem and the access to compute that I have.

You can get the code for inference and training on Github vagmi/jevlite. The trained QLoRA adapter is on HuggingFace at vagmi/jev-lite. It implements the TypeSafe System One API, and I tested it against their python client.

Let us see how this works.

What does an LLM do?

Given some text, it produces a score for every possible next word fragment (token).

A token is usually a word or word-fragment. For e.g., Gemma 4 knows 262,144 of them. It is called the model’s vocabulary. Given the tokens of this phrase The capital of Canada is, it produces 262,144 numbers, one per token, and the number attached to the token corresponding to Ottawa is higher than for all the other tokens.

These raw scores are called logits. They are not probabilities they are unbounded numbers, positive or negative, where bigger means that the token associated with it is a more likely choice to be the next.

So, to write text, the inference engine does this in a loop:

1) read all tokens and run it through the model
1) score every token
2) pick one near the top
3) stick it to the end
4) Go back to step one until you get <|eos|>.

The <|eos|> token is the end-of-sequence token and in most models is literally represented this way.

So the big idea for jevlite is that we only need to steps 1 through 3. There is no loop. We can simply score a constrained set of choices of tokens we choose.

Step 1: Prompt Engineering

The first step is to come up with a prompt strategy. We set it up in such a way that it always has to answer with one token. In jevlite, it looks like this.

Read the state and answer the question with one option letter.

<state>
Help! My payouts have been failing for 3 days.
</state>

Question: How frustrated is the customer?
Levels, lowest to highest:
A. Calm
B. Frustrated
C. Very angry
Answer:

The prompt stops at the : character.

For a multiple-choice question, we describe the options so the model knows what each label means:

Read the state and answer the question with one option letter.

<state>
Help! My payouts have been failing for 3 days.
</state>

Question: Which team should handle this?
Options:
A. billing — Payments, invoicing, refunds
B. technical — Bugs, outages, integrations
Answer:

Since we are fine tuning an instruction trained model, the description gives it semantic understanding.

Step 2: Read the probabilities

The prompt gets broken down into a series of tokens. For the first example, the model sees a series of 67 numbers and we ask it to predict the score of the number in the 68th position.

   [ 60]  236780  'C'
   [ 61]  236761  '.'
   [ 62]   16707  ' Very'
   [ 63]   23186  ' angry'
   [ 64]     107  '\n'
   [ 65]    7925  'Answer'
   [ 66]  236787  ':'
   [ 67]  <-- Hey model, what comes here?

In code that is one function call and one array index:

# one pass over 67 tokens
output = model(input_ids=tokens)     
# read the score of all the 262,144 tokens at the 68th position
logits = output.logits[0, -1]

After the weights are loaded to memory this just takes 20ms.

Step 3: Derive our scores from logits

We asked a question with three options, so we are only interested in those three tokens A, B and C.

letters = [562, 603, 565]            # token ids for " A", " B", " C"
scores  = [logits[i] for i in letters]
# [26.6250, 29.3750, 27.5000]

Those are real numbers from the model, for the ticket above. B (“Frustrated”) scores highest at 29.375.

Here are the five highest-scoring tokens in the entire vocabulary at that position:

   603  ' B'   +29.3750
   565  ' C'   +27.5000
   562  ' A'   +26.6250
   622  ' D'   +25.1250
   645  ' E'   +24.7500

Before fine tuning, the valid option was in the 6th position.

Step 4: Turn scores into probabilities

Logits are unbounded and don’t sum to anything. To get probabilities we use softmax, which is less intimidating than it sounds. Two rules:

  1. Bigger score → bigger probability, and the relationship is exponential, so a score 2 points higher is a lot more likely, not a little.
  2. The results must add up to 1.

Here it is in our friendly python.

import math

def softmax(scores):
    biggest = max(scores)
    weights = [math.exp(s - biggest) for s in scores]
    total = sum(weights)
    return [w / total for w in weights]

Run it on our three scores:

>>> softmax([26.6250, 29.3750, 27.5000])
[0.0525, 0.8215, 0.1260]

Let us walk through this:

A (Calm) B (Frustrated) C (Very angry)
raw score 26.6250 29.3750 27.5000
minus the biggest −2.75 0 −1.875
exp() of that 0.0639 1.0000 0.1534
divided by the total (1.2173) 0.0525 0.8215 0.1260

The model is 82% sure the customer is “Frustrated”, with 13% leaning toward “Very angry” and 5% on “Calm”.

Notice the slieght of hand here. We ran softmax over our choices, not over the entire vocabulary of 262,144 tokens. This is what makes Jev feel magical and when they say it cannot hallucinate. The math makes it impossible.

Building Typesafe Primitives: Choices, Scores and Noul

Typesafe has modeled three primitives. It has choices, scores and a boolean operation called noul. Here we just do an inference trick.

A choice: “Pick one”

We turn the score into a probability.

def choice_answer(options, probs):
    best = max(range(len(probs)), key=lambda i: probs[i])
    return {"choice": options[best],
            "probabilities": dict(zip(options, probs)),
            "confidence": probs[best]}

For our routing question the model returned:

{"choice": "billing",
 "probabilities": {"billing": 0.5622, "technical": 0.4378},
 "confidence": 0.5622}

Look at that split. Failing payouts genuinely could be either team, and the model says so 56/44. A system that only ever gave you the winning label would have said “billing” and hidden the fact that it was nearly a coin toss.

A score: “How much”

Here the options are ordered Calm < Frustrated < Very angry. Here we do not just treat it as a classification problem but the mood on a scale. They are in-between frustrated and very angry. So it needs to be a bit towards the right. This bit of code with the expected level takes these probabilities and nudges it towards very angry.

def expected_level(probs):
    return sum(level * p for level, p in enumerate(probs))
>>> expected_level([0.0525, 0.8215, 0.1260])
1.0735

The Noul: “Is it true?”

Two options, true and false, and the answer is simply the probability of true:

def noul_answer(probs):
    # options are always ["true", "false"]
    return probs[0]
>>> noul_answer([0.9399, 0.0601])
0.9399

There is no “choosing” here at all. A result of 0.94 means yes. A result of 0.38 means “probably not, but don’t bet on it”.

In Conclusion

It was fun building this. With vLLM, I can push 600 inferences a second with nothing running on my RTX 4090. This field never ceases to surprise me.