Jev vs GPT at chess: dead even, at a twelfth of the price

Kidding. Mostly. What I actually think is overkill for most people is seeing Jev drive a vehicle, play Doom and do stuff. I mean, yes, it's a sort of "classifier", but does it truly hold up?

So, this article is not a technical deep dive into Jev, but rather an experiment to see if it actually stands. That said, the core use case of Jev, as a classifier / router is phenomenal, which you'll see soon.

So, I vibe coded a chess simulator and put Jev against GPT. To keep things fair (and under budget), I picked gpt-5.4-mini to go up against Jev.

Wiring it up

I grabbed Claude and asked it for a very basic chess game, plus a way for Jev and GPT to face each other. To keep things fair (the best I could), I made sure the state going in and the output coming out were the same for both Jev and GPT.

So, I wired the state as follows:

{
  "model": "jev-latest",
  "state": {
    "board": {
      "e8": "black king",
      "d8": "black queen",
      "e4": "white pawn" /* …every occupied square */,
    },
    "color": "black",
    "history": ["1. white pawn e2-e4 (e4)"],
    "comments": "",
  },
  "questions": {
    // which piece? one option per piece you still have
    "selected_piece": {
      "type": "choice",
      "criteria": {
        /* your 16 pieces */
      },
    },
    // where to? one option per square on the board
    "move_to_block": {
      "type": "choice",
      "criteria": {
        /* all 64 squares */
      },
    },
  },
}

The state is just the facts: what's on each square, whose turn it is, the moves so far, and a comments field where I could talk back to the model. The output was two answers: which piece to move, and where to move it.

Everything wired up, time to test it

And guess what? Jev failed miserably. The first few moves already looked almost random, and after that every other move was illegal. GPT threw in a few illegal moves too, but Jev was unplayable after the first couple of turns, even after I spelled it out that the previous move X was illegal.

This is when, for me, all those self-driving simulations and Jev playing Doom started to look gimmicky. And to some extent it was clear that Jev is nothing without the underlying homies.

Looking back, it kinda makes sense. Jev answers the questions in a request independently, so the square question had no clue which piece I'd picked. After 1.e4, Black has 16 × 64 = 1,024 piece-and-square pairs, and only 20 of them are legal moves. On top of that, figuring out where a knight can go from a list of squares is multi-step geometry. And retrying didn't help either: it went for Nc3-a6 four times in a row. Commitment, I guess.

Giving it the legal moves

To make sure no illegal move could ever happen, I changed a few things around. The state stayed as it was, with one extra thing: a list of every legal move you can play this turn.

Here's the whole turn. My code builds it once, and both models get it:

{
  "prompt": "You are chess grandmaster, its your turn, look at the board and history and color and play the next move. The options are all your legal moves.",
  "state": {
    "board": {
      "f8": "black bishop",
      "g8": "black knight",
      "e7": "black king",
      "e2": "white king" /* …every occupied square */,
    },
    "color": "white",
    "history": [
      /* … */ "30. white king d1-e2 (Ke2)",
      "30... black king d7-e7 (Ke7)",
    ],
    "comments": "",
  },
  "question": "Which move do you play?",
  // every legal move, and nothing else
  "options": {
    "e2d1": "Kd1: white king e2-d1",
    "e2e3": "Ke3: white king e2-e3",
    "e2f1": "Kf1: white king e2-f1",
    "a2a3": "a3: white pawn a2-a3",
    // …
  },
}

Each model then gets that exact object, just packed into its own API's format. Jev takes the state as is, with the move as a typed Choice question and the legal moves as its criteria:

{
  "model": "jev-latest",
  "state": {
    "board": {
      /* … */
    },
    "color": "white",
    "history": [
      /* … */
    ],
    "comments": "",
  },
  "questions": {
    "move": {
      "type": "choice",
      "instructions": {
        "prompt": "You are chess grandmaster, …",
        "question": "Which move do you play?",
      },
      "criteria": {
        "e2d1": "Kd1: white king e2-d1",
        "e2e3": "Ke3: white king e2-e3" /* …every legal move */,
      },
    },
  },
}

GPT gets the same prompt as its instructions, and the same state and question as its input:

{
  "model": "gpt-5.4-mini",
  "instructions": "You are chess grandmaster, …",
  // sent as a JSON string, unpacked here so you can read it
  "input": {
    "state": {
      "board": {
        /* … */
      },
      "color": "white",
      "history": [
        /* … */
      ],
      "comments": "",
    },
    "questions": {
      "move": {
        "question": "Which move do you play?",
        "options": {
          "e2d1": "Kd1: white king e2-d1",
          "e2e3": "Ke3: white king e2-e3" /* …every legal move */,
        },
      },
    },
  },
  "text": {
    "format": {
      "type": "json_schema",
      "name": "move",
      "strict": true,
      "schema": {
        /* the answer schema, below */
      },
    },
  },
}

Same prompt, same state, same question, same list of moves. Only the envelope changes. And it's not just me pinky-promising it's fair: there's a test in the repo that fails the moment the two drift apart.

test("choice mode: Jev and GPT get the same prompt, state, question and legal moves", () => {
  const step = moveStep(playSan(inMode("1vgpt"), "e4"));
  const jevQ = jevRequest(step).questions.move;
  const body = gptRequest(step, DEFAULT_GPT_MODEL);
 
  assert.equal(body.instructions, jevQ.instructions.prompt);
  assert.deepEqual(JSON.parse(body.input), {
    state: jevRequest(step).state,
    questions: {
      move: { question: jevQ.instructions.question, options: jevQ.criteria },
    },
  });
 
  // GPT answers { move, reason }: the move limited to the same legal moves.
  const { format } = body.text;
  assert.deepEqual(
    format.schema.properties!.move.enum,
    Object.keys(jevQ.criteria),
  );
});

By doing this, it was certain that either model would make a legal move 100% of the time, even if it's random or ridiculous. My code works out the legal moves, and the model only has to pick one. Which, not so coincidentally, is exactly the classifier / router job Jev is built for.

Two nodes: choice and score

And I also added two nodes: the first is choice, and the second is score, which is probability based. Both players in a game always play the same node, so the output they're held to is the same too.

Choice is the one you've seen above: "Which move do you play?", pick one of the options. Jev hands back its pick, a probability for every option, and a confidence:

{
  "answers": {
    "move": {
      "choice": "e2d1",
      "probabilities": {
        "e2d1": 0.66,
        "e2e3": 0.05,
        "e2f1": 0.04,
        "a2a3": 0.02
      },
      "confidence": 0.65
    }
  },
  "usage": { "input_tokens": 1916, "output_tokens": 210 }
}

GPT, being a chat model, has to say why as well. Its answer is locked to a strict JSON schema, and move is an enum of the exact same option keys Jev got, so it can't wander off-list:

// the schema
{
  "type": "object",
  "properties": {
    "move": { "type": "string", "description": "Which move do you play?", "enum": ["d7c7", "d7e7" /* …every legal move */] },
    "reason": { "type": "string", "description": "Why you chose this move." }
  },
  "required": ["move", "reason"],
  "additionalProperties": false
}
 
// the answer
{
  "move": "d7c7",
  "reason": "The king is in the center with only a few legal king moves. Kc7 steps out of the d-file, avoids immediate tactics on the king, and keeps the position compact and defensible."
}

Score flips it around. Instead of picking one move, the model rates every legal move from 0 (a blunder) to 4 (an excellent move), and the highest score gets played. The state is still the same; the question changes, and the scale comes along. For Jev, that's one Score question per legal move, all in a single request, and one score back per move:

// one of these per legal move, all in the same request
"a2a3": {
  "type": "score",
  "instructions": {
    "prompt": "You are chess grandmaster, its your turn, look at the board and history and color and play the next move. The options are all your legal moves.",
    "question": "How good is this move for you? Rate it on the scale.",
    "move": "a3: white pawn a2-a3"
  },
  "criteria": [
    "a blunder: loses material or the game",
    "a weak move: gives the opponent the advantage",
    "an ordinary move: keeps the position about equal",
    "a good move: improves your position",
    "an excellent move: wins material, gives mate or is clearly the best"
  ]
}
 
// the answer: a score per move, the highest one is played
{ "answers": { "d2g5": { "score": 2.26 }, "d2e3": { "score": 2.25 }, "d2f4": { "score": 2.22 } /* … */ } }

GPT gets the same question and the same scale, and has to come back with a number for every single legal move, plus its reasoning:

{
  "type": "object",
  "properties": {
    "scores": {
      "type": "object",
      "description": "How good is this move for you? Rate it on the scale. One score per move, from 0 (a blunder: loses material or the game) to 4 (an excellent move: wins material, gives mate or is clearly the best); decimals allowed.",
      "properties": {
        "d2g5": { "type": "number" },
        "d2e3": { "type": "number" } /* …one per legal move */,
      },
      "required": ["d2g5", "d2e3" /* …every legal move */],
      "additionalProperties": false,
    },
    "reason": {
      "type": "string",
      "description": "Your overall reasoning: why the best move scores highest.",
    },
  },
  "required": ["scores", "reason"],
  "additionalProperties": false,
}

One fun detail: Jev never writes a single word. The "reason" you see next to Jev's moves in my logs is my app summarizing its numbers. GPT writes its own reasons, and we'll get to how much those are worth.

So, was it any good?

But honestly, the games were not that impressive. Jev was definitely fast and cheap, as we all knew, but it was kinda at the same intelligence level as gpt-5.4-mini. GPT hung its queen in two of my recorded games (6…Qxd5?? 7.Qxd5, a free queen), and Jev once gave its queen away to a lone king. Grandmasters, both of them.

So, I upped the game. I added a simulator that runs 10 games in parallel and then analyzes how each model performed.

The results were almost a tie. Most games were either a draw or hit the max move limit, and Jev won one game, same as gpt-5.4-mini.

A few things stood out.

Nobody could close out a game. 7 of the 10 games ran into the 60-move limit, one was a threefold repetition, and only 2 ended in checkmate. Jev finished +0.8 material per game on average, but a lead you can't convert is just a draw with extra steps. My favorite: in game 3, GPT promoted a pawn to a queen on move 60 of 60, and the game ended in a draw right there. Impeccable timing.

Jev loves a check. It gave 69 checks against GPT's 40, and made more captures too (128 vs 112). In my earlier games, plenty of those checks just handed the checking piece over. Check first, think later.

GPT's reasons sound great, but don't trust them. Every GPT move comes with a neat little explanation, and sometimes it's describing a different board. The best one explains that its king is in check from the other king, which, last I checked, isn't a thing. Jev writes no text at all, so at least it can't lie to you.

Jev is way cheaper, and faster. Those 10 games cost $0.092 for Jev against $1.15 for GPT, about 12× cheaper, and Jev took 2.2 seconds a move against GPT's 3.9.

Ten games isn't a benchmark, so take the tie with a pinch of salt. But it paints a pretty clear picture.

To conclude

It's best that we don't overhype Jev, but at the same time, failing to realize its true potential would be naive.

Give it a list of options and it picks between them fast and dirt cheap. That's the router, and here it held its own against GPT for about a twelfth of the price. Ask it to come up with the options itself, and it falls apart. So give it the right job, and give it the underlying homies.

Catch the demo live here, and the code here.