Key Notes
- Claude Fable 5.1 scored 79.7 out of 100 in HalachaBench’s provisional comparison of 22 models on 60 Hebrew questions drawn from rabbinical ordination exams.
- Fable helped write the grading rubrics and GPT-6 Astra served as the main judge, while no rabbi has yet validated the rubrics or answers.
- The project says its scores are not ordination exam grades, and the results do not clearly separate Fable from second-place Gemini 4 Argon.
Claude has taken an early lead in an unusual AI contest: answering questions drawn from Israel’s rabbinical ordination exams. Fable 5.1 scored 79.7 out of 100 in HalachaBench, ahead of Gemini 4 Argon at 78.2. The result tests a demanding combination of Hebrew, religious sources and detailed reasoning, rather than the coding tasks that dominate many model comparisons.
The independent project’s November 2 results cover 22 models answering 60 questions from five Chief Rabbinate of Israel exam papers. Each model made three attempts at every question, in Hebrew, without books or web access. The numbers are provisional: no rabbi has yet reviewed the grading rubrics or marked the answers.
Six Models Scored Above 60
Behind Fable and Argon, GPT-6 Astra scored 73.3, GPT-6.1 Sol reached 72.8 and Claude Opus 5.5 scored 69.1. Gemini 3.8 Flash followed at 64.4. Those six crossed 60 on the project’s scale, but HalachaBench explicitly says its scores are not semicha exam grades and do not establish that a model would pass ordination exams.
The questions span family purity, marriage, agricultural commandments associated with the Land of Israel, ritual baths, and prayer and blessings. Answering well requires more than recognizing a familiar religious term. A useful response needs to connect the question with relevant sources, explain the reasoning and distinguish the conclusion from other accepted views.
The eight locally run models were far behind the leaders, with Muse Glimmer 30B highest at 10.5. That finding applies to the tested configurations: local runs used compressed weights, and some models exhausted their output budgets before producing an answer. It does not establish a ceiling for every open-weight model or a full-precision deployment.
The gap makes this an interesting specialist test. A model that writes fluent Hebrew may still omit a necessary source or skip a step that matters to the answer. Conversely, a response can sound less polished while identifying the right issue. Evaluation needs to capture that difference rather than rewarding confidence alone.
The Leading Model Helped Write the Marking Rules
Fable 5.1 drafted and revised the rubrics used to assess answers. Other models reviewed them, but Fable retained a role in deciding which suggested changes to accept. GPT-6 Astra, another contestant, served as the main judge. There is no official answer key behind those AI-generated marking rules.
That arrangement creates a possible evaluation bias, not proof that the leaderboard is wrong. A rubric writer may favor the structure or detail its own answers tend to supply. Independent expert marking is needed to establish whether the scores reflect the subject accurately and treat different approaches fairly.
The narrow lead over Argon also deserves restraint. The project’s statistical comparison does not clearly separate the two models. Exam questions can recur across sittings, and possible exposure to similar questions during training has not been ruled out. Closed-book testing prevents a fresh search; it cannot erase material a model may already have encountered.
A Benchmark Score Is Not Rabbinic Authority
HalachaBench plans rabbinic review of the rubrics and a random sample of answers to check and adjust the AI grading. Until that work is complete, the ranking measures performance under an experimental marking system. It offers no human baseline supporting claims that the models found the task easier than well-prepared candidates.
The distinction is practical. An AI-generated answer can help someone locate an issue to study, but fluent explanation and a high benchmark score do not establish the authority or reliability of a religious ruling. The project itself warns against using these results as permission to rely on a model for halachic decisions.
For AI research, the next useful result will be whether qualified reviewers agree with the automated scores. If they do, HalachaBench could provide a valuable test of specialized Hebrew reasoning. If they identify missing sources, disputed interpretations or grading inconsistencies, those findings would be just as informative about what today’s models still fail to understand.
Disclaimer: AIstify is an independent media brand owned and operated by NuvexMedia LLC, publishing news, research, and insights on artificial intelligence, emerging technologies, automation, and related industries. NuvexMedia LLC invests in and collaborates with companies across the AI, technology, software, and digital innovation sectors. These relationships do not influence AIstify’s editorial coverage, and the publication maintains full editorial independence to provide accurate, timely, and objective information. © 2026 NuvexMedia LLC. All rights reserved. This content is for informational purposes only and should not be considered legal, tax, investment, financial, or other professional advice.
