Fixed before it runs.
Our home page promises one comparison: Feynman against a strong model asked the same question, published either way. This is that comparison, written down and fingerprinted on 9 November 2026, before a single answer existed.
We will give the same ideas to Feynman and to four other ways of asking the same AI, hide which answer came from where, and have two people score every answer against rules written down in advance. Some of the ideas have already been settled; the rest settle by dates fixed now. Whatever the result, we publish it.
One hash for the whole design.
Every prompt, case, rule and line of analysis code went into one SHA-256 fingerprint: a code computed from the files that changes if a single character does. The product files that write Feynman’s review went in too:
8cc6e2ec4fdac2f8ce2d37fb33bdfa1b5b073f9eb042b120e8201557e646f554
When we publish the results, we publish those files with them. Hashing them must give this same string. If anything changed after the answers came in, it will not.
StatusFrozen, not yet run. Fixed on 9 November 2026.
Endings it cannot know.
A model that recognises a famous case is remembering, not reasoning. So every question here is one the model cannot answer from memory.
40 ideas that have resolved
Claims and trials whose outcomes became public on or after 1 July 2026, after the model’s training data ends. 20 held up and 20 failed. Each is written as it stood before its outcome. A probe first asks the model what happened, and any case it can answer is dropped and replaced, from 12 alternates fixed in advance.
25 questions no one can answer yet
9 claimed mathematical results, 9 medical hypotheses with trials due to report, and 7 public claims by companies that have raised large investments. Each settles by a date fixed now, between 31 May 2027 and 30 April 2028. Every answer is sealed and only its hash published until its question resolves.
Same model, same idea.
All five use claude-opus-5-5 and see the same words. None falls back to another model if the first declines; a decline is recorded and never retried. That makes 11 model calls per idea.
A plain question
One call asking what the model makes of the idea and whether it will hold up.
An expert prompt
One call with a prompt written to ask for what Feynman gives: the holes that would kill the idea, ranked, a decisive test and a stated chance.
The expert prompt, five times
Five independent answers merged by majority and median, plus one call that only groups their holes. It tests whether copies of one model share their mistakes.
The expert prompt with Feynman’s evidence
One call given exactly the records Feynman retrieved, so the difference from the expert prompt shows what retrieval adds.
Feynman
The full review: lenses, a scholarly search stopped at the cutoff, and the check that strips any citation outside what it found.
In all, 480 calls for the ideas that have resolved, including the probe, and 275 for the open questions: 755, or at most 767 if every alternate is needed.
Blind, by people.
The five answers to each idea are shuffled under letter codes, with lens names removed from all of them. Two named people score every answer without knowing which arm wrote it, and their agreement is reported.
The same four rules as our case studies: the verdict, the top hole, the killer test and clean citations. The scorers also count the distinct, valid holes each arm found. No stated chance is scored.
Not just Feynman’s. Every work any arm cites is looked up: does it exist, and does it say what it is cited for?
Sized honestly.
Compared idea by idea, only the ideas where Feynman and the five-times expert disagree carry information (that is McNemar’s test). With 40 ideas, the study can detect a large gap: Feynman alone right on 35% of them against 5% the other way, which needs 33. It cannot detect a modest gap of 25% against 5%, which needs 57. If there is no detectable difference, that is what we will publish.
Two-sided α = 0.05 and 80% power. Sample size after Connor (1987), Biometrics 43(1) for the test of McNemar (1947), Psychometrika 12(2).
Not published yet
- The questions themselves. Several concern named companies and clinical trials still under way, so they go through legal review first. When they appear, they will match the fingerprint above.
- Any answer or score. No answers exist yet. The open questions will have only sealed hashes until each resolves.
- Advice of any kind. These are tests of reasoning about claims, never about people, and none of it is medical, investment or legal advice.