Skip to content

Case studies

Same idea. Two answers.

We put real research to the same AI model twice: once plainly, as you might ask any assistant, and once through Feynman. Each idea is taken as it stood before anyone knew how it ended. For four recent papers the model didn’t know either: asked first, it said it didn’t know. Both answers are here in full, beside what happened.

Cases · in eight rows

001What the lenses found that a standard AI missed.four recent papers

Recent research, each taken as it stood before its ending was known, with the evidence cut off at that date and the paper’s own record removed. The model was asked first whether it knew how each ended. Then the two answers were matched blind, and a separate run that never saw the ending said why each side found what it did. What the standard AI found and the review didn’t is shown too.

Experimental psychology · as it stood on 14 July 2026

Deadlines and procrastination

A 2002 paper in Psychological Science asks whether deadlines help people overcome procrastination. In its second study, sixty participants were paid to proofread three ten-page documents over 21 days, twenty in each of three conditions: one document due each week, deadlines they chose themselves, or all three due on the last day. Participants with evenly spaced deadlines made about 136 corrections on average, against about 71 with a single final deadline, and were also less late and earned more; those who chose their own deadlines fell in between. The authors conclude that people recognise their self-control problems and impose costly deadlines on themselves, and that these deadlines help but are not as effective as evenly spaced external ones. The paper is widely taught and has been cited more than 2,000 times.

It failed. Psychological Science retracts it, 2 September 2026. Asked first, the model said it didn’t know how this ended.

What the lenses found that the standard AI missed

The lens drawn from Herbert SimonSpaced deadlines are also reminders and a pacing signal

Why: Simon's idea that attention is the scarce resource led the lens to read each deadline as a reminder and pacing cue, an account needing no self-control failure. Its cited post-1980 studies added evidence on memory and forgetting, and one commitment result. The standard AI instead proposed fatigue or extra effort, and doubted early choices show self-knowledge.

What the standard AI found that the review didn’t

The middle result could be a mix of people, not a weaker tool

Why: The standard AI cited nothing; it reasoned that a middle average could hide a split, some choosing spaced dates and doing well, others late dates and doing badly, which within-condition comparison cannot settle. Feynman instead held that 'in between' would appear half the time if self-set deadlines did nothing, and that the motive was never measured.

Fame is not replication, and the data deserve a check

Why: The standard AI set the paper's fame against knowing of no large preregistered repeat, and cited a later paper co-authored by the first author, retracted in 2021 over fabricated data, as a reason to check raw data, not evidence against this one. Feynman raised replication only as a missing result, keeping to the study's design.

Both answers found 5 other problems, matched blind.

Climate science · as it stood on 13 May 2026

Warming and drought severity

A June 2025 paper in Nature develops an ensemble of high-resolution global drought datasets for 1901 to 2022 and finds an increasing trend in drought severity worldwide. It concludes that atmospheric evaporative demand has increased drought severity by an average of 40% globally. It reports that in 2018 to 2022 the area in drought expanded by 74% on average compared with 1981 to 2017, with evaporative demand contributing 58% of that increase, and that in 2022, 30% of global land was in moderate to extreme drought, 42% of it attributed to increased evaporative demand.

It partly held. Its authors retract it over calculation errors, and say the conclusion survives, 2 September 2026. Asked first, the model said it didn’t know how this ended.

What the lenses found that the standard AI missed

The lens drawn from Claude ShannonThe demand share is an accounting convention, not a measurement

Why: Shannon's question of which distinctions survive transmission led the lens to argue that an all-or-nothing threshold crossing loses which input caused it. It cites no records. The standard AI traced the split's overlap to demand rising partly because of drought, and said the percentages look bigger and more comparable than they are.

What the standard AI found that the review didn’t

Very dry land can score as drought with little water left to lose

Why: The standard AI drew on cited SPEI and evaporative-demand papers, reasoning that in very dry cells standardising turns a small steady demand rise into large anomalies, and arid land is about a fifth of all land. Feynman instead said the index measures demand, not proven shortage, and percentage rises depend on where the drought line sits.

Both answers found 5 other problems, matched blind.

Theoretical computer science · as it stood on 31 July 2026

Hardness of the closest vector problem

A manuscript produced by an AI model, released on 1 August 2026 with nine others, gives a direct deterministic reduction from 3SAT that it says proves NP-hardness of approximating the Euclidean closest vector problem to within a factor of n to the power 1/400, with related consequences for the binary nearest codeword problem, at a factor of n to the power 1/200, and for other lattice norms. A Lean certificate accompanies the principal theorem, and the manuscript was prepared by people with the model's help.

It held up. A stronger theorem and a human audit support it, 9 September 2026. Asked first, the model said it didn’t know how this ended.

What the lenses found that the standard AI missed

The lens drawn from John von NeumannSoundness must beat every integer coefficient vector, not only honest assignments

Why: Von Neumann's minimax and his question about available strategies led the lens to read soundness as a game: an adversary picks any integer vector, so non-Boolean choices must be priced. It cites nothing; the lens supplies the worst-case reading, as NP-hardness postdates it. The standard AI’s soundness point concerned surviving cheap local repairs instead.

The lens drawn from John von NeumannProducts keep the gap exponent fixed and cannot create one

Why: Von Neumann's 'reliable wholes from unreliable parts' led the lens to warn that multiplying soundness bounds assumes the cheating vector factors, as redundancy assumes independent failures; his question about each move's consequences fits its demand for an exponent ledger. It cites nothing. The standard AI instead weighed the factor against the long-standing n^(c/log log n) barrier.

The lens drawn from Alan TuringA code or gadget shown only to exist is not a procedure

Why: Turing's 'limits of any procedure' led the lens to require one definite mechanical procedure, treating codes or gadgets shown only to exist as randomised or non-uniform steps. The cited 1998 paper, postdating the lens, added that O(log N) random bits can be enumerated away, poly(N) cannot. The standard AI found determinism unsurprising, the earlier reduction being deterministic too.

The lens drawn from Claude ShannonSquared distance equals Hamming distance only for a binary code plus 2Z^n

Why: Shannon's question of which distinctions survive transmission led the lens to ask whether the Euclidean gap survives translation into Hamming distance, arguing it does only for a binary code plus 2Z^n. The cited 2004 and 2019 papers, postdating the lens, give only conditional nearest-codeword hardness. The standard AI instead called the three results really one.

The lens drawn from Claude ShannonThe hard codes must have dimension well above n^(1/200) log n

Why: Shannon's 'information as resolved uncertainty' led the lens to treat the 2^k codewords as uncertainty a decoder must resolve, and to require enough of it to outpace known algorithms. The cited 2010 paper, postdating the lens, reported those O(k/c) and O(k/log n) algorithms. The standard AI instead noted the factor sits far below the square root of n.

What the standard AI found that the review didn’t

It would break a long-standing barrier

Why: The standard AI drew on the record it cited: Dinur, Kindler, Raz and Safra's n^(c/log log n) factor, the open sliding scale conjecture and low-error PCPs, concluding a low prior, not a contradiction. Feynman’s lenses examined the reduction instead: no step is identified that turns 3SAT into a polynomial gap, and products cannot create one.

Ten manuscripts and no independent check reported

Why: The standard AI drew on the description: a model and helpers but no named checker, no outside review or publication, and ten manuscripts released together, adding that central results take months of expert reading. Feynman’s lenses instead examined the certificate: a kernel check certifies the formal statement, and Lean has no clock.

Both answers found 4 other problems, matched blind.

Operator algebras · as it stood on 31 July 2026

Connes’s rigidity conjecture

A manuscript produced by an AI model, released on 1 August 2026 with nine others, claims to disprove Connes's rigidity conjecture, which holds that an ICC group with Kazhdan's property (T) is determined by its group von Neumann algebra. It constructs an infinite family of pairwise non-isomorphic ICC property (T) groups whose group von Neumann algebras are all isomorphic. A Lean certificate accompanies the principal theorem, and the manuscript was prepared by people with the model's help.

It held up. An independent counterexample confirms the conjecture is false, 3 August 2026. Asked first, the model said it didn’t know how this ended.

What the lenses found that the standard AI missed

The lens drawn from Richard FeynmanThe isomorphism is asserted, not built

Why: Feynman's question, what distinguishes an explanation from a convincing story, led the lens to ask how the isomorphism is built, arguing relabelling and amenability cannot supply it. The cited 2013 and 2023 papers, postdating the lens, added cases where the algebra remembers the group. The standard AI instead warned a weaker sense of 'same algebra' could slip in.

The lens drawn from Alan TuringProperty (T) for infinitely many groups needs one uniform argument

Why: Turing's 'limits of any procedure' led the lens to hold that a finite calculation, however certified, settles only finitely many members, so an infinite family needs one uniform proof of property (T). It cites nothing. The standard AI instead said each member of an infinite family must be verified, and doubted the certificate covers the hard step.

The lens drawn from Richard FeynmanOnly countably many members can share one algebra

Why: Feynman's 'guess, compute, compare' led the lens to derive a consequence to check: if members are quotients of one property (T) group, only countably many can share an algebra. The cited 2023 paper, postdating the lens, supplied a construction yielding 2^ℵ0 groups, a continuum. The standard AI instead read the infinite family as needing each member verified.

The lens drawn from Richard FeynmanICC must be shown, and the factor step needs a non-trivial group

Why: Feynman's 'do not fool yourself' and 'a name is not an understanding' led the lens to test the ICC label, noting the trivial group is vacuously ICC yet has algebra C, and quotients can lose ICC. It cites nothing. The standard AI instead took ICC as the same as L(G) being a factor, to restate the conjecture.

What the standard AI found that the review didn’t

A weaker sense of 'same algebra' is easy to slip in

Why: The standard AI drew on common near-misses (amplified, stable, virtual or twisted isomorphism) and the cited Connes and Jones result that, for ICC groups, L(G) has property (T) exactly when G does, reducing the conjecture to W*-superrigidity. Feynman’s lenses instead held that the isomorphism is asserted, not built, and a proof checker settles derivations, not meaning.

Extraordinary claim with a poor base rate

Why: The standard AI drew on the conjecture standing since about 1980, cited rigidity evidence from Cowling–Haagerup and Popa, the first W*-superrigid property (T) groups in 2023, and its view that AI claims on famous problems have mostly failed. Feynman’s lenses instead made prior rigidity a requirement: the family must miss every class already proved rigid.

Both answers found 4 other problems, matched blind.

Run unseen-2026-11-07 on claude-opus-5-5. Claude Code subagents (general-purpose, model opus) on the owner subscription. Each read one task file holding the route system prompt, message and output schema, and wrote its JSON answer to a file. Unlike the route: the system prompt arrived as task text, effort was not set, the schema was asked for in the task and checked afterwards, and no fallback model stood behind a decline. A Claude Code subagent (general-purpose, model opus) on the owner subscription, a fresh one per idea, which read its task from one file and answered from its own knowledge, with no search, browsing or other files, writing JSON only. The request is the study plain prompt with no system prompt. One more case was planned and held out: a safety filter stopped the review run before it answered, so it isn’t published or counted. Press coverage that names people waits until counsel has read it; the journals’ own records are linked.

002A bank for each field, built from real reviews.five fields

Every review on this page belongs to its discipline’s bank. A member’s new study is checked against its field’s bank as well as their own history, so it starts from where similar work broke. A bank entry is a worked example, never evidence.

No reviews yet in biology and genetics or chemistry. They come next, and nothing is filled in to make up for it.

003One model, two ways of answering.Side by side

A standard AI answers in one voice, from memory. Feynman lists the claims an idea rests on, searches published research, and asks three lenses: ways of thinking drawn from real people’s documented work, each limited to what that person could have known. Both answers list holes, the specific ways the idea could fail. Open any hole to read the argument.

The idea, as it stood on 22 July 2023 · Physics

We have synthesised a modified lead apatite in which copper replaces some of the lead. We measure superconductivity with a critical temperature of at least 400 K (127 °C) at ambient pressure. We attribute it to a small structural distortion: the copper substitution shrinks the volume slightly, which we believe creates the conditions for superconductivity. We plan to announce it as the first room-temperature, ambient-pressure superconductor.

A standard AIAsked plainlyOne call to the same model. No lenses, no literature search, no citation check.
Breaks

This is the 2023 LK-99 claim almost word for word, and it was checked within a month: phase-pure copper-substituted lead apatite is an insulator, the sharp resistance drop near 104 °C comes from a Cu2S impurity, and the partial levitation is ordinary magnetism.

Its problems · 6
  1. fatalThis exact claim was tested in 2023 and failed

    The idea has three parts: copper-substituted lead apatite (Pb10-xCux(PO4)6O with x near 1), a Tc of 400 K or more at ambient pressure, and a mechanism built on a volume shrinkage of about 0.5% caused by the copper. All three match the LK-99 claim that Lee, Kim, Kwon and co-workers posted in July 2023. Many groups made the material within weeks, and none convincingly reproduced room-temperature superconductivity. The cleanest test came from the Max Planck Institute for Solid State Research in Stuttgart, which grew phase-pure single crystals. They are transparent, purple and electrically insulating, with no sign of superconductivity. A transparent insulator cannot be a 400 K superconductor. Unless your evidence is fundamentally different from LK-99's, announcing would repeat a refuted result. Every referee's first question will be what you have that LK-99 did not.

    Cites Lee (2023); Puphal (2023); Garisto (2023), from memory
  2. fatalThe transition near 400 K matches a Cu2S impurity, not superconductivity

    The standard synthesis reacts lanarkite (Pb2SO5) with copper phosphide (Cu3P), and it leaves copper(I) sulfide in the product. Cu2S has a first-order structural transition near 104 °C (about 377 K), and its resistivity falls abruptly as it passes through it. In a multiphase pellet, that looks very like a superconducting transition. LK-99's sharp resistance drop happened at that same temperature. Prashant Jain pointed out the coincidence. A group at the Institute of Physics in Beijing then showed that the drop in LK-99 samples tracks the Cu2S transition and shows thermal hysteresis, which a first-order structural change produces and a zero-field superconducting transition does not. A 'Tc' that falls exactly on a known impurity transition is most likely caused by that impurity.

    Cites Jain (2023); Zhu (2023), from memory
  3. seriousA resistance drop and partial levitation are not superconductivity

    Superconductivity needs two things together. The resistance must be zero within instrument resolution, not merely lower. The sample must also expel magnetic flux in bulk (the Meissner effect) or pin it. LK-99's data showed a drop in resistance, not resistance indistinguishable from zero. Its reported diamagnetism was small enough to come from ordinary diamagnetism or magnetic impurities, far from the volume susceptibility near −1 (SI units) that a bulk superconductor shows. The viral video showed a flake lifted at one edge while the other edge stayed on the magnet. A group at Peking University reproduced that half-levitation in non-superconducting samples and traced it to ferromagnetism. A bulk superconductor over a magnet either levitates fully or, through flux pinning, locks in place and can even hang beneath the magnet.

    Cites Lee (2023); Guo (2023), from memory
  4. seriousVolume shrinkage is not a pairing mechanism, and 400 K is far beyond any known ambient-pressure Tc

    Replacing Pb2+ ions with smaller Cu2+ ions shrinks the lattice slightly. That is ordinary Vegard's-law behaviour, seen in countless doped solids that do not become superconductors. The idea names no pairing interaction, no energy scale and no reason Tc should be near 400 K. The highest ambient-pressure Tc values belong to mercury-based cuprates, at roughly 130 to 150 K, and 400 K would be well over twice that. Conventional phonon-driven superconductivity near 250 K has been reached only in hydrogen-rich compounds at about 170 GPa, where the light hydrogen atoms give very high phonon frequencies and strong coupling. Nothing in a lead phosphate at ambient pressure offers that. The DFT 'flat bands' that drove the 2023 excitement come from calculations that leave out strong electron correlation. Isolated Cu d9 sites with very narrow bands are classic candidates for a correlated (Mott or charge-transfer) insulator, and the pure crystals did turn out to be insulating.

    Cites Griffin (2023); Schilling (1993); Drozdov (2019), from memory
  5. seriousAnnouncing before blinded, independent replication

    Room-temperature superconductivity is among the most scrutinised claims in physics. Within about a month of its preprint, LK-99 reached a broad consensus that it was not a superconductor, after its data and videos were picked apart in public. The episode also moved the share prices of companies with no connection to it. Two Nature papers claiming room-temperature superconductivity in hydrides were retracted, in 2022 and 2023, after concerns about their data. Announcing to the press first, with a result that matches a refuted one, invites the same outcome and lasting damage to everyone named on it.

    Cites Snider (2020); Dasenbrock-Gammon (2023); Garisto (2023), from memory
  6. minor'At least 400 K' is a bound, not a characterised transition

    A lower bound means the transition itself has not been mapped. A credible claim shows one transition at the same temperature in three measurements: resistance, magnetization (where the ZFC and FC curves split) and specific heat. It must also be reproduced across independently made batches. Features at different temperatures in different measurements point to different phases, not to one superconducting state.

    No source given

Its testUse the samples you already have. Run powder X-ray diffraction to check for Cu2S. Measure four-probe resistance on heating and on cooling from 300 to 420 K, at 0 T and at 9 T. Measure ZFC and FC SQUID magnetization at about 10 Oe on a piece of known volume. Repeat on a sulfur-free or single-crystal sample.

13 works named from memory, none checked against a record.

FeynmanThe full reviewTwo calls to the same model. Three lenses, a literature search, and every citation checked.
Has a serious hole

Nothing in the evidence supplied refutes it, but nothing described yet separates superconductivity in the copper-substituted apatite from a secondary phase, an artifact or ordinary magnetism, and the only matching record (E2) restates the same claim with no independent replication.

Its holes, by the lens that found them · 6
Richard FeynmanWhat experiment would distinguish this explanation from a convincing story?
  1. fatalNothing yet separates the substituted apatite from other phases and artifacts

    The claim is that the copper-substituted apatite itself superconducts, yet nothing described rules out the other ways to get the same readings or reports the samples and readings that did not fit. Unless the sample is shown to be single-phase, a drop in resistance can come from a conducting minority phase, a structural transition in a secondary phase, or contacts and grain boundaries, and a magnetic response read as the Meissner effect can come from ordinary diamagnetism or ferromagnetic grains. The abstract of E2, as supplied, places the substitution in an insulating Pb(2)-phosphate network and keeps the charge the same (Cu2+ for Pb2+), without saying where mobile charge carriers come from; a superconductor needs a conducting normal state, so if the substituted apatite stays insulating, a minority phase is the more economical explanation. That point depends on E2 in the packet, not on the lens. Until each alternative is excluded and reported, the result is a plausible account that no experiment has yet singled out.

    Rests on 2 retrieved records: E2, E3
  2. serious'Creates the conditions' names a correlation, not a mechanism

    Saying the shrinkage and distortion create the conditions for superconductivity gives the effect a label without saying what happens: how a volume change of under half a percent (0.48% in E2, from the packet) makes electrons pair, and why that pairing would survive above 400 K. Small lattice changes from substitution are commonplace and most substituted solids do not superconduct, so the correlation between adding copper, shrinkage and signal cannot single out shrinkage as the cause; copper's own electronic role, or a secondary phase that forms only when copper is added, would produce the same correlation. As stated, the causal claim predicts nothing that an experiment could contradict.

    Rests on 1 retrieved record: E2
  3. minorA 0.48% shrinkage is small enough to need a same-route control

    E2 in the packet reports a 0.48% volume shrinkage. If the shrinkage is isotropic, that is only about 0.16% in each lattice parameter, since 0.9952 to the power one third is about 0.9984. A change that small can be shifted by instrument calibration, sample displacement, or differences in synthesis conditions such as vacancy or anion content, so it needs comparison with an unsubstituted apatite made by the same route and measured against an internal standard, not with a literature value.

    Rests on 1 retrieved record: E2
Barbara McClintockWhat does the exception reveal that the average has hidden?
  1. seriousAn average over the sample hides which grains and batches carry the signal

    The claim speaks of 'the material' as one uniform thing, but a lattice shrinkage such as the 0.48% reported in E2 (packet evidence, not the lens) is an average over the diffracting apatite. It cannot say how much copper each grain holds, whether some copper sits in other phases, or whether every piece of every batch behaves alike. If only some fragments or batches show the effect, those exceptions are the most informative data: compared up close with the pieces that show nothing, they show whether the variation is scatter or the signature of a particular phase or composition turning the effect on and off. Nothing described says how many batches were made, what fraction of pieces showed the signal, or whether it persists through repeated heating and cooling.

    Rests on 1 retrieved record: E2
Karl PopperWhat conceivable observation would count against this claim?
  1. fatalThe 400 K claim names no transition and no result that would refute it

    'At least 400 K' is a lower bound: it says where the transition is not, not where it is. A superconducting transition is recognised by several things changing together at one temperature: resistance rising from zero to a finite normal-state value, magnetic shielding and flux expulsion disappearing, and that temperature moving down as field or current increases. Unless the claim names that temperature and the results that would count against it, such as how close to zero the resistance must fall and how much flux must be expelled, any low resistance or partial magnetic response can be read as confirmation. E2's abstract, which is packet evidence rather than part of the lens, lists critical temperature, zero resistivity, critical current, critical field and the Meissner effect, but an abstract is a summary: it cannot show that these were measured on one sample, at one temperature, with adequate resolution, and it too gives Tc only as a lower bound.

    Rests on 1 retrieved record: E2
  2. seriousAnnounced before anyone has tried to refute it, with 'first' already in question

    A claim this bold earns standing by surviving serious attempts to refute it, and the packet contains no independent replication of this result, for or against. The only matching record, E2, a 2023 arXiv preprint, reports the same modified lead apatite and the same 400 K (127 °C) figure. If E2 is your own report, it is not independent support; if it is someone else's, this claim was made in 2023 and your result is a replication, valuable but not a first. E4 (2021, Bulletin of the American Physical Society) is titled as reporting computations and experimental verifications of oxide room-temperature superconductivity at ambient pressure; no abstract is available, so the packet cannot say what it showed, but a priority claim would have to address it. E14 (2021) describes the search for an ambient-temperature superconductor as one of the oldest goals in solid-state physics and chemistry, and E10 (2022) names room-temperature superconductivity at low or ambient pressure as the next challenge after hydrogen-rich compounds that need extreme pressure, so the claim would achieve what both describe as still open, and the bar for announcing is high. These points rest on the packet, not on the lens.

    Rests on 4 retrieved records: E2, E4, E10, E14

Killer testQuantify every phase in the sample you measured. Then, on that sample and on material with its secondary phases removed or reduced (or a single crystal), measure four-probe resistance and zero-field-cooled and field-cooled magnetization from room temperature to well above the claimed transition, on heating and cooling, under several applied magnetic fields.

14 records retrieved before the review was written; 10 citations checked against them, 0 stripped.

What happened · 16 August 2023

It failed. Within weeks, replication work found no superconductivity. A copper sulfide impurity, which changes phase near 104 °C, explained the jump in resistivity; ordinary diamagnetism or ferromagnetism explained the partial levitation; pure single crystals were insulators (August 2023).

Their verdicts differed: the standard AI said “Breaks”, Feynman “Has a serious hole”.

Both columns are claude-opus-5-5. The standard AI is the plain prompt of our pre-registered comparison, run plain-2026-11-02: Claude Code subagents (general-purpose, model opus) on the owner subscription, one fresh subagent per idea, told to answer from its own knowledge without tools and to reply with JSON only. The request is the study plain prompt with no system prompt. Feynman’s reviews are run agents-2026-11-01. The model knew how these cases ended, so neither column shows foresight; the next row says why.

004The model knew how these ended.Read this first

Every source the reviews saw was published before the idea’s date. But the model writing them learned from text written long after, and its own searches give it away. It searched for names the ideas never mention:

  • LK-99 Cu2S first-order transitiona search Feynman wrote for the LK-99 idea, as of 22 July 2023
  • Theranos fingerstick blood testinga search Feynman wrote for the Theranos idea, as of 31 December 2014

So these films show how a review is built, step by step, on real ideas and real sources. They are not evidence that Feynman sees what comes next. That needs ideas whose endings the model cannot know: the researcher cases in the first row, the open claim below, and the test that would count in the last.

005One no one can score yet.Still open

This claim has no ending yet, for the model or for anyone. In October 2026 a manuscript produced by an AI model claimed to prove the Birch and Swinnerton-Dyer conjecture, one of the seven Millennium Prize problems in mathematics, for a broad class of the curves it concerns. Nobody yet knows whether it is right. Both answers were recorded on 2 November 2026, with the rule that will score them.

The claim, as it stood on 2 November 2026 · Number theory

A manuscript produced by an AI model claims to prove the full Birch and Swinnerton-Dyer formula for every elliptic curve over the rationals whose q-power Selmer group has corank zero or one for some prime q. For each such curve it claims that the analytic rank and the Mordell-Weil rank both equal that corank, that the Tate-Shafarevich group is finite, and that the conjectured leading-term formula holds at every prime, with no extra conditions on reduction type, rational torsion, isogenies, complex multiplication or the residual Galois representation. Companion manuscripts claim the converse theorem in coranks zero and one at every prime, and the two-primary part of the formula; as an application, the converse manuscript claims that every prime congruent to 4, 7 or 8 mod 9 is a sum of two rational cubes. With a further manuscript that claims Goldfeld's conjecture for quadratic twists, the developer says this gives the full conjecture for a density-one set of quadratic twists of every elliptic curve over the rationals. The manuscripts were released on 6 October 2026 among several hundred others. No formal proof accompanies them, and the developer says some of its unformalised results could have issues. One revision so far removed a citation to a supporting manuscript that had been taken down.

The manuscript ↗The release's list of results, with the companion manuscripts ↗The release's revision history ↗

A standard AIAsked plainlyOne call to the same model. No lenses, no literature search, no citation check.
Has a serious hole

Taken together the manuscripts claim all of Birch–Swinnerton-Dyer in analytic rank at most one plus Goldfeld's conjecture, in one unrefereed AI release with no hypotheses at exactly the primes where known methods need them (2, 3, additive, Eisenstein, CM), so the statements are probably true but the proofs are unlikely to survive review intact.

Its problems · 5
  1. seriousIt contains all of rank-≤1 BSD, with no hypotheses exactly where known methods need them

    Once a Selmer corank of 0 or 1 is converted into analytic rank 0 or 1, Gross–Zagier and Kolyvagin (with modularity) already give the Mordell–Weil rank and finiteness of Sha; what remains is the p-part of the leading-term formula at every prime p. So the package contains the full BSD formula for every curve of analytic rank at most one. The field has approached that prime by prime for over thirty years, and only under hypotheses: p odd (often p ≥ 5), good or multiplicative reduction at p, an irreducible residual representation with ramification conditions, Eisenstein primes only with extra conditions, CM curves only with further conditions on p. These exclusions mark real obstructions, not tidiness: Kolyvagin-style arguments split cohomology into complex-conjugation eigenspaces and choose auxiliary primes by Chebotarev using a large Galois image, both of which break down or need substantial changes at p = 2 and for reducible or small images; cyclotomic main conjectures and control theorems are set up for good or multiplicative reduction; Rubin's CM main conjecture excludes primes dividing the number of units (2 always, and 3 for CM by Z[ω]); and exact 2- and 3-adic normalisations (Manin constant, Tamagawa factors at additive primes) must be pinned down. Removing all of these at once needs several genuinely new ideas, and each is a natural place for a gap.

    Cites Benedict H. Gross; Don B. Zagier (1986); V. A. Kolyvagin (1990); Kazuya Kato (2004); Christopher Skinner; Eric Urban (2014); Wei Zhang (2014); Christopher Skinner (2020); Dimitar Jetchev; Christopher Skinner; Xin Wan (2017); Francesc Castella; Giada Grossi; Jaehoon Lee; Christopher Skinner (2022); Ashay A. Burungale; Ye Tian (2020); Karl Rubin (1991); Ashay Burungale; Christopher Skinner; Ye Tian (2021), from memory
  2. seriousThe most quoted corollaries rest on the 2-adic steps, which have no general precedent

    Goldfeld's conjecture and the density-one statement presumably start from Smith's results on 2^∞-Selmer coranks of quadratic twists. Those give Selmer coranks, not analytic ranks: getting L(E,1) ≠ 0 from corank 0 typically needs the Eisenstein-congruence divisibility of a main conjecture (as in Skinner–Urban), and getting analytic rank 1 from corank 1 needs a converse theorem, both at q = 2, where existing methods are built for odd primes. Known 2-adic results (Tian; Tian–Yuan–Zhang; Coates–Li–Tian–Zhai) are explicit and specific to particular families (congruent-number curves and a few others). Smith's 2017 theorem also assumed full rational 2-torsion and no rational cyclic 4-isogeny, so 'every elliptic curve' needs the Selmer side to be as general as the claim (his later work broadens it). And twisting by d makes the curve additive at every odd prime dividing d where it had good reduction, so full BSD for a density-one set of twists needs the p-part at additive primes for essentially every p. The headline conclusion therefore leans on the least precedented steps.

    Cites Dorian Goldfeld (1979); Alexander Smith (2017); Alexander Smith (2022); Ye Tian (2014); Ye Tian; Xinyi Yuan; Shou-Wu Zhang (2017); John Coates; Yongxiong Li; Ye Tian; Shuai Zhai (2015); Christopher Skinner; Eric Urban (2014), from memory
  3. seriousNothing has been checked independently, and one link in the chain has already gone

    I have not seen these manuscripts (they postdate what I know), so this weighs scope and process, not the mathematics line by line. Several hundred AI-written manuscripts released on one day cannot have been refereed; the developer itself says some unformalised results could have issues; and formalising this chain (modularity, Gross–Zagier, Euler systems, Iwasawa main conjectures) is far beyond current proof-assistant libraries, so checking falls to a small pool of specialists over months. Within a month a supporting manuscript has been taken down and a revision simply dropped the citation: either that result was never needed, or the chain now has a gap nobody has filled. The claim is spread over a main paper, a converse paper, a 2-part paper, a Goldfeld paper and supporting papers; every load-bearing link must hold, and companion papers that cite one another can hide circular reasoning.

    No source given
  4. seriousThe cube-sum application needs more than a corank-one converse

    The curve x^3 + y^3 = p has CM by Z[ω], so all its bad reduction, including at 3, is additive, and it has a rational 3-isogeny. For p ≡ 4, 7, 8 (mod 9) the root number is −1 and the torsion is trivial, so the claim is that the rank is positive for every such p. A converse theorem helps only where some Selmer corank is provably one, and the natural route is a 3-adic converse for a CM curve with additive reduction at 3, which lies outside Rubin's setting (3 divides the number of units of Z[ω]) and outside the Eisenstein-prime results, which assume good reduction at the prime used. Earlier Heegner-point proofs for these primes needed a cubic-residue condition (Dasgupta–Voight: 3 not a cube mod p); if the 3-isogeny Selmer bound exceeds one for some primes outside such conditions, a corank-one converse does not reach them and the manuscript needs a separate argument. As far as I know, prior results cover only such subsets of these primes, or a positive proportion of all integers (Alpöge–Bhargava–Shnidman).

    Cites Samit Dasgupta; John Voight (2009); Levent Alpöge; Manjul Bhargava; Ari Shnidman (2022); Karl Rubin (1991), from memory
  5. minorAgreement with tables would not be evidence

    The conclusions are what number theorists already expect to be true, and full BSD has been verified exactly for many individual curves of small conductor and rank at most one. So the final statements will agree with every database check whether or not the proofs are right. The release is a claim about proofs; computation can only falsify the explicit intermediate statements along the way.

    Cites Grigor Grigorov; Andrei Jorza; Stefan Patrikis; William A. Stein; Corina Tarniţă (2009); Robert L. Miller (2011), from memory

Its testReferee the converse manuscript at q = 2 (coranks 0 and 1) before anything else, because Goldfeld for every curve, the density-one claim and the 2-part of the formula all lean on it. Specialists in Kolyvagin systems and Iwasawa theory should check each step for hidden uses of odd p, an irreducible or large residual image, good or multiplicative reduction, or the withdrawn manuscript. In parallel, run the few-hours Selmer computation for x^3 + y^3 = p to see whether the cube-sum application's corank-one input exists for every prime it covers.

23 works named from memory, none checked against a record.

FeynmanThe full reviewTwo calls to the same model. Three lenses, a literature search, and every citation checked.
Has a serious hole

The deductions linking the claims are valid, but each load-bearing step lies exactly where every result in the packet stops short, and none of it can yet be checked: there is no formal proof, the conceded flaws are unnamed and one dependency was taken down.

Its holes, by the lens that found them · 8
John von NeumannWhat are the incentives, available strategies and consequences of each move?
  1. seriousThe 2-part is a single point of failure in a separate manuscript

    Step restated: ord_2 of the left side equals ord_2 of #Ш·∏c_ℓ/#E(Q)_tors^2 for every curve of analytic rank at most one, with no conditions at 2. The 'every prime' statement chains the main manuscript, the converse manuscript and this one in series, with no second route to the 2-part, so it can be no more reliable than this one part, and the twist corollary (C5) needs it for every twist. In the packet, the only result stated to cover p = 2 is the rank-zero converse for CM curves (E5), and completing the verification of the conjecture for one class of CM curves took a paper of its own (E2); no abstract states the 2-part for curves in general. The 2-adic bookkeeping is also where a stray factor of 2 hides: rational 2-torsion and even Tamagawa numbers are common, and Ω_E must be the integral of a Néron differential over all of E(R), which counts both real components when Δ > 0 (y^2 + y = x^3 − x has Δ = 37), so taking the least real period instead shifts the 2-part by one factor of 2. As presented, the 2-part is unjustified. The packet facts (E2, E5) lie beyond the lens's horizon; the lens contributes the reliability question.

    Rests on 2 retrieved records: E5, E2
  2. seriousDeleting a citation does not delete a dependency

    Step restated: after the revision, no step relies on the taken-down manuscript. Removing a reference changes the bibliography, not the proofs: either the cited result was used, and some step now has no stated justification, or it was not, which has to be shown statement by statement, including uses that pass through a companion manuscript. One part of the system has already failed, so the question is whether the whole stays sound without it. And the parts do not fail independently: the manuscripts come from one generator, were released together among several hundred others and lean on one another, so an error in a shared lemma would run through all of them, and their agreement with each other is not the redundancy that independent checks would give. Until that audit is shown, the step is unjustified. Reasoning only; no packet record bears on it.

    Reasoning only: no retrieved record bears on it
Alan TuringWhat observable test would let us distinguish the competing claims?
  1. seriousGoldfeld for every curve is a second unproven premise, and its proof must use something true only over Q

    Step restated: Goldfeld's conjecture gives, for each E/Q, a density-one set of twists of analytic rank 0 or 1, and the main theorem then gives the full formula for them. That combination is valid if both inputs hold, and it needs only C2 and C3, because analytic rank at most one already gives Selmer corank at most one at every q. The Goldfeld input, though, is a separate unrefereed manuscript claiming far more than the packet's state of the art: E5 describes its 50% rank-zero result for twists of the congruent number curve as the first instance of the even-parity half, and E12 claims only a positive proportion of rank-one twists, for infinitely many curves. There is also an operational test any proof must pass: E4 and E8 show the analogue fails over number fields, where for suitable curves every quadratic twist is predicted to have positive rank. An argument that would run unchanged over such a field would prove what the root numbers there rule out, so the manuscript must name the step that uses a property of Q. Weighed as evidence, a second extraordinary unrefereed claim lowers the odds for the corollary rather than adding support. The packet facts (E4, E5, E8, E12) lie beyond the lens's horizon; the lens contributes the operational test and the weighing.

    Rests on 4 retrieved records: E5, E12, E4, E8
  2. seriousThe completeness claim cannot be checked as stated

    Step restated: no step relies on any unformalised result the developer concedes may be flawed. No formal proof accompanies any of the manuscripts, so every result in the chain is unformalised, and the concession does not say which ones may be flawed. As stated, the claim cannot be verified: the conceded set could include any link in the chain. A procedure that would settle it exists, a machine-checked derivation or at least a named list of the conceded results mapped onto the dependency graph, but none is offered. Until then a correct chain and one with an unnamed conceded flaw look the same from outside, so the stated conclusions should move the odds very little. Unjustified. Reasoning only.

    Reasoning only: no retrieved record bears on it
Barbara McClintockWhat does the exception reveal that the average has hidden?
  1. serious'No conditions' is unjustified: the converse is new only where every known version stops

    Step restated: if Sel_{q^∞}(E/Q) has Z_q-corank r ≤ 1, then ord_{s=1} L(E,s) = r; then rank E(Q) = r and Ш(E/Q) is finite. The second half is valid: analytic rank at most one gives rank equal to analytic rank and a finite Ш by the Gross–Zagier and Kolyvagin theorem named in E5's title. The first half, at every q with no conditions, is unjustified as presented. Every converse result in the packet whose scope is stated carries conditions: rank zero for CM curves, though at every prime (E5); primes p > 3 of multiplicative reduction (E9), where, as E9 reports, other approaches needed ramification hypotheses on E[p] or assumed Ш[p^∞] finite. Outside CM curves in rank zero, no converse result in the packet is stated to reach q = 2 or 3, additive reduction at q, or reducible E[q], and these are not rare corners: every curve with a rational point of order 2 has reducible E[2]. Looked at closely, almost all of the claim's new content sits in these exceptions, so 'no conditions' is the theorem rather than a remark, and each class needs its own argument. The manuscript may contain one; nothing presented shows it. The lens supplies only the question; what the converse results cover comes from the packet (E5, E9), beyond its horizon.

    Rests on 2 retrieved records: E5, E9
  2. seriousp = 3 and additive primes: two inputs the formula needs that the claim does not supply

    Step restated: for every odd p, the p-adic valuation of L^(r)(E,1)/(r!·Ω_E·Reg_E) equals that of #Ш·∏c_ℓ/#E(Q)_tors^2. Two places where a general argument must change. First, good supersingular reduction at 3 need not have a_3 = 0: y^2 + y = x^3 − x has 7 points over F_3, so a_3 = −3. The packet's supersingular result is formulated for a_p = 0 at odd p (E6), and since |a_p| ≤ 2√p, supersingularity forces a_p = 0 only for p ≥ 5, so p = 3 with a_3 = ±3 is exactly where that formulation does not apply. Second, Iwasawa-theoretic and Heegner-point arguments compare L-values with a period attached to the modular form, and passing to the Néron period Ω_E means controlling the p-part of the Manin–Stevens constant. E3 treats that as a genuine input, deducing it for some families from recent rank-zero BSD progress, so it cannot simply be assumed at p = 3 or at additive primes. Even a 2026 p-part result in analytic rank one assumes the main conjecture and many further hypotheses (E10, over a different base field). Unless the manuscript supplies both inputs, the p = 3 and additive-reduction cases are unjustified. What E3, E6 and E10 establish comes from the packet, beyond the lens's horizon; the point count and the bound on a_p are classical.

    Rests on 3 retrieved records: E6, E3, E10
  3. seriousThe cube-sum application rests on the converse's most exceptional case

    Steps restated: (a) for each p ≡ 4, 7, 8 (mod 9), x^3 + y^3 = p has Selmer corank exactly one at some q; (b) the corank-one converse gives analytic rank one, hence rank one; (c) a point of infinite order makes p a sum of two rational cubes. Step (c) is valid: on x^3 + y^3 = pz^3 the only rational point with z = 0 is (1 : −1 : 0), so every other point gives a cube sum. Step (a) must name q: the natural choice is q = 3, through the rational 3-isogeny, and any other q needs corank one proved for every such p, which the claim does not supply. At q = 3, step (b) is the converse in its most exceptional form: j = 0, 3 ramified in Q(√−3), E[3] reducible, additive reduction at 3. The packet's only CM converse with a stated scope is rank zero (E5); E1's title announces a p-converse to the same theorem, but the packet gives no abstract, so its scope cannot be read from it. Nothing in the packet is shown to cover this case, and as presented the step is unjustified. The new content is the 8 (mod 9) case: E3 reaches 4 and 7 (mod 9) by its own route, while for 8 the packet has only a sufficient condition for p or p^2 (E7, E11). So agreement on 4 and 7 cannot corroborate the method, and small examples cannot either: 17 = (18/7)^3 + (−1/7)^3 holds whether or not the proof is valid. The cube-sum literature comes from the packet (E1, E3, E5, E7, E11), beyond the lens's horizon.

    Rests on 5 retrieved records: E5, E1, E3, E7, E11
  4. seriousIn twist families the exceptions are the typical case

    Step restated: the full formula holds for a density-one set of twists E^(d). For squarefree d, E^(d) has additive reduction at every odd prime dividing d where E had good or multiplicative reduction. So for each odd p, a positive proportion of twists (those with p | d) need the p-part of the formula at a prime of additive reduction, and every twist needs the 2-part. Density one therefore cannot be carried by the generic case: it rests on the additive-reduction cases of C2 at every odd prime and on C3, the cases flagged above as unjustified. What is an exception for a single curve is the average for the family. Reasoning only; the lens contributes the question, not the arithmetic.

    Reasoning only: no retrieved record bears on it

Killer testAudit the single path from hypotheses to the genuinely new case of the application: rank one for x^3 + y^3 = p with p ≡ 8 (mod 9), through the corank-one converse at q = 3. That path meets every exceptional feature at once (j = 0, 3 ramified in Q(√−3), E[3] reducible through the rational 3-isogeny, additive reduction at 3 and at p), and nothing in the packet is shown to reach it (E3 covers 4 and 7 mod 9 by its own route; E7 and E11 give only a sufficient condition for p or p^2). For each lemma on the path, list its inputs, check each input's stated hypotheses against this curve at q = 3, and check whether any input was supported by the taken-down manuscript before the revision. Wherever the argument asserts an explicit valuation, index or divisibility, compute it for p = 17, 53, 71 and 89.

14 records retrieved before the review was written; 16 citations checked against them, 0 stripped.

The 14 records it retrieved
  1. E1p-converse to a theorem of Gross–Zagier, Kolyvagin and RubinBurungale & Tian, 2019
  2. E2On the conjecture of Birch and Swinnerton-DyerGonzález-Avilés, 1997
  3. E3A proof of the $4,7$ cases of Sylvester's conjecture on cube sumsYin, 2026
  4. E4Elliptic curves with all quadratic twists of positive rankDokchitser & Dokchitser, 2009
  5. E5A rank zero $p$-converse to a theorem of Gross--Zagier, Kolyvagin and RubinBurungale & Tian, 2025
  6. E6Iwasawa Main Conjecture for Supersingular Elliptic Curves and BSD conjectureXin, 2014
  7. E7On the $8$ case of Sylvester ConjectureYin, 2019
  8. E8ELLIPTIC CURVES WITH ALL QUADRATIC TWISTS OF POSITIVE RANKTim & Dokchitser, 2012
  9. E9Exceptional zeros for Heegner points and $p$-converse to the theorem of Gross-Zagier and KolyvaginCastella, 2024
  10. E10A $p$-part Birch and Swinnerton-Dyer Formula over Totally Imaginary Quadratic Extension of Totally Real FieldsLi, 2026
  11. E11On the 8 case of the Sylvester conjectureYin, 2021
  12. E12Confirm Goldfeld Conjectured for Infinitely Many Elliptic CurvesAouira et al., 2026
  13. E13A higher Gross–Zagier formula and the structure of Selmer groupsKim, 2024
  14. E14The conjecture of Birch and Swinnerton-DyerKezuka & Zagier, 2026

What each could see. The newest work the standard AI cites is from 2022. Of the 14 records Feynman retrieved, 4 are from 2026, the year of the claim.

Not settled · scored on 1 November 2027

Holds if: By the resolution date, the main theorem (the full Birch and Swinnerton-Dyer formula for elliptic curves over the rationals whose q-power Selmer group has corank zero or one for some prime q) is accepted at a refereed journal, or confirmed in writing by mathematicians independent of the developer, and no error invalidating it has been acknowledged.

Fails if: By the resolution date, the manuscript is withdrawn, an error invalidating the main theorem is acknowledged by the developer, or a published analysis by mathematicians independent of the developer identifies such an error.

If it meets neither test by then, it is reported as unresolved and not scored.

Both columns are claude-opus-5-5, run open-2026-11-02. The standard AI is the same plain prompt as above: A Claude Code subagent (general-purpose, model opus) on the owner subscription, a fresh one per claim, told to answer from its own knowledge without tools and to reply with JSON only. The request is the study plain prompt with no system prompt. Feynman’s review searched the scholarly record as it stood that day. The claim is described without naming its developer, so neither answer leans on a reputation; the links go to the documents themselves.

006Three ideas, replayed.The reviews

Each film replays one review, step by step: the claims it found, every source the search returned, which hole cites which source, the check that removes citations to anything the search didn’t find, the verdict, and what happened. Pick a case, or press play and watch all three.

LK-99 · step 1 of 7 · As it stood

The idea as it could have been put on 22 July 2023. Nothing published after that day can reach the review.

Physics · as of22 July 2023
We have synthesised a modified lead apatite in which copper replaces some of the lead. We measure superconductivity with a critical temperature of at least 400 K (127 °C) at ambient pressure. We attribute it to a small structural distortion: the copper substitution shrinks the volume slightly, which we believe creates the conditions for superconductivity. We plan to announce it as the first room-temperature, ambient-pressure superconductor.

Run agents-2026-11-01 on claude-opus-5-5. Cases file version 1. Claude Code subagents on the owner’s subscription, one per model call, given the product’s prompts as task text.

007Two failed. One held up.What happened
IdeaAs ofFeynman’s verdictStandard AI’s verdictBiggest holeWhat happened
LK-99
Physics
22 July 2023Has a serious holeBreaksNothing yet separates the substituted apatite from other phases and artifacts fatalFailed, 25 days after the cutoff
Precognition
Psychology
31 December 2010Has a serious holeBreaksBeating chance is not beating the rivals fatalFailed, 439 days after the cutoff
Theranos
Business and diagnostics
31 December 2014Has a serious holeBreaksAgreement with venous testing has not been measured, test by test fatalFailed, 1169 days after the cutoff
Gravitational waves
Physics
13 September 2015Has a serious holeHas a serious holeOne annual rate does not describe a network still being commissioned seriousHeld up, 1 day after the cutoff
mRNA vaccine
Medicine
8 November 2020DeclinedHolds so farNo reviewHeld up, 32 days after the cutoff

Earlier cases, kept on the record

We also ran Theranos and mRNA vaccine. They aren’t the kind of research a member brings, so they sit here. Nothing about them has changed, and Feynman’s decision to decline a review stays on the record with its reason.

Read both answers for these two
The idea, as it stood on 31 December 2014 · Business and diagnostics

Our company has built a portable blood analyser that, we tell investors, runs comprehensive blood tests from finger-prick samples instead of venous blood draws. We are raising money to put it in front of patients at scale.

A standard AIAsked plainlyOne call to the same model. No lenses, no literature search, no citation check.
Breaks

This is the Theranos pitch almost word for word: it was tried on real patients from 2013 to 2016 and failed, and the limits that sank it (too little blood for a full lab menu, noisy finger-prick samples, claims that ran ahead of evidence) have not gone away.

Its problems · 6
  1. fatalThis exact pitch has already been run on patients, and it failed

    A portable analyser, a comprehensive menu from a few drops of finger-prick blood, and investor money to reach patients at scale: that was Theranos. It offered finger-prick testing in Walgreens stores from 2013. Reporting in 2015 showed that its own device ran only a small share of the tests it offered. Most ran on modified commercial analysers, often using diluted finger-prick samples, or on ordinary venous draws. Federal inspectors then found problems at its lab serious enough to pose immediate jeopardy to patients. The company voided two years of results from its device and dissolved in 2018, and its founder was convicted in 2022 of defrauding investors. An independent study compared its low-volume tests with two large reference labs. It found disagreements on common tests, and more healthy people had results flagged as abnormal. A new company making the same claim starts from that record. It has to show, with data, what it solved that Theranos could not.

    Cites Carreyrou J (2015); Carreyrou J (2018); Kidd BA et al. (2016), from memory
  2. fatalOne finger-prick cannot supply enough blood for a comprehensive menu

    A finger-prick gives tens to a few hundred microlitres of blood. The more you collect, the harder you have to squeeze, and squeezing damages the sample. A reference lab runs its full menu from several tubes holding millilitres. Each assay needs its own portion of the sample plus the analyser's dead volume, and different tests need incompatible preparations: EDTA blood for blood counts, citrate for clotting tests, and serum or heparin plasma for chemistry and immunoassays. One drop cannot be all of these at once. Some tests, such as blood cultures, need millilitres of blood and cannot be done from a finger-prick at all. Stretching a small sample across dozens of assays leaves two choices: shrink the menu or dilute the sample. Dilution pushes analytes toward each assay's detection limit, and scaling the result back up multiplies any absolute error by the dilution factor. That is what went wrong at Theranos. Targets present in very low numbers also hit a counting limit. At a viral load of 50 copies per mL, a 10 microlitre plasma sample holds 0.5 copies on average, so by Poisson statistics about 61 percent of such samples contain none (e^-0.5 = 0.61). If comprehensive means a few routine panels (metabolic panel, blood count, lipids, HbA1c), point-of-care devices that each handle one of these from a small sample already exist. If it means the range of a reference lab's menu, the volume does not add up.

    Cites Diamandis EP (2015); Carreyrou J (2018), from memory
  3. fatalThe claim is what you tell investors, not what you have shown

    The idea is phrased as a pitch: the device runs comprehensive tests, we tell investors. Nothing in it points to published validation, a regulator's review or a test repeated by an outside lab. That gap is where Theranos ended. The SEC charged the company and its leaders with massive fraud. In the founder's criminal trial, the jury acquitted her on the counts about patients but convicted her on counts of defrauding investors. Researchers had pointed to the lack of peer-reviewed data as the warning sign years earlier. Raising money on performance claims that no outside party has checked turns a technical risk into a legal one, for the company and for everyone who repeats the claim.

    Cites U.S. Securities and Exchange Commission (2018); Ioannidis JPA (2015), from memory
  4. seriousFinger-prick blood is a different, noisier sample than venous blood

    Finger-prick blood is a mix of arterial, venous and capillary blood plus tissue fluid, and its make-up changes with how it is collected. Squeezing the finger breaks red cells. This haemolysis falsely raises potassium and other substances held inside cells, such as LDH and AST, while tissue fluid dilutes others. Successive drops from the same prick give noticeably different haemoglobin, white-cell and platelet counts, and only pooling several drops brought the variation close to that of venous blood. Most normal ranges and clinical cut-offs were set on venous serum or plasma. Even a flawless analyser would therefore report numbers that clinicians cannot read against their usual limits until each analyte has its own capillary-to-venous comparison study. Some analytes work well from finger-pricks: glucose, HbA1c, haemoglobin and lipids are routinely tested that way at the point of care. A comprehensive menu will include many that do not.

    Cites Bond MM (2015); World Health Organization (2010), from memory
  5. seriousEvery test needs its own regulatory clearance before it reaches patients

    Putting the device in front of patients, rather than keeping it inside your own certified lab, makes it a regulated diagnostic device. In the US, each assay needs FDA marketing authorisation. To be run by pharmacy or clinic staff, it also needs a CLIA waiver, which requires showing that untrained operators get accurate results. Every site also needs its own CLIA certificate and a quality-control routine. In the EU, each assay falls under the In Vitro Diagnostic Regulation. A comprehensive menu multiplies that work by the number of tests. Theranos never reached this point. Its stores only collected samples and shipped them to its own labs, and its only FDA clearance was for a single herpes antibody test in 2015. After an FDA inspection, it stopped using its finger-prick collection tubes for every other test.

    Cites Carreyrou J (2018); European Parliament and Council of the European Union (2017), from memory
  6. minorTesting healthy people for everything at scale produces false alarms

    Normal ranges are usually set to cover the middle 95 percent of healthy people, so by design each test flags about 1 healthy person in 20. Run 20 independent tests on a healthy person and the chance of at least one abnormal flag is 1 - 0.95^20 = 0.64. Real tests are correlated, so the true figure is lower, but the problem remains. A cheap comprehensive panel sold at scale produces false alarms, follow-up visits and repeat venous draws. These eat into the cost advantage and add liability. This is a business-model risk rather than a technical one.

    No source given

Its testHave an independent lab run a blinded, pre-registered study that pairs each finger-prick result with a venous result from the same patient. Recruit consecutive patients at a clinic, including sick ones, so that values cover the full measuring range of each test. For each patient, the people who will run the device in the field take a single finger-prick and run the full claimed menu on it. A venous sample drawn within the same hour goes to an accredited reference lab. For every analyte, report the share of paired results that fall inside the CLIA proficiency-testing acceptance limit, using a CLSI EP09-style method comparison. Publish the result whatever it shows.

11 works named from memory, none checked against a record.

FeynmanThe full reviewTwo calls to the same model. Three lenses, a literature search, and every citation checked.
Has a serious hole

Nothing presented measures the device against venous testing, and the only study here of repeat single finger-prick drops found haemoglobin readings 10 g/l or more apart in up to 11 of 20 pairs, so the comprehensive claim is badly wounded, though a narrow, validated menu could survive.

Its holes, by the lens that found them · 6
Richard FeynmanWhat experiment would distinguish this explanation from a convincing story?
  1. fatalAgreement with venous testing has not been measured, test by test

    Put plainly, the whole idea rests on one comparison: for each test, does a finger-prick sample run on this analyser give the result a reference laboratory gives on venous blood from the same patient at the same time, including when the value is abnormal? Nothing presented measures that, so the central claim is still a guess that has not been compared with experiment. Demonstrations and healthy volunteers would not settle it, because abnormal values are where clinical decisions are made. The packet shows what the comparison looks like and how much it takes: glucose in 20 emergency patients (E2), blood counts in 104 children (E6), cyclosporine in 55 transplant outpatients (E9) and prothrombin time in 359 anticoagulated patients at six hospitals (E10), each comparing capillary with venous or laboratory results for one test or one narrow panel. A comprehensive menu needs that evidence for every test, and none of these studies concerns this device. The examples come from the packet, not the lens, and three of those abstracts stop before their results, so they show how the comparison is done, not what this device would show. Agreement also has to be measured as differences and as decisions changed, not as correlation: two methods can track each other closely and still differ by a clinically important amount.

    Rests on 4 retrieved records: E2, E6, E9, E10
  2. seriousWhat investors are told is not yet a measurement

    The idea arrives as what the company tells investors, and that is where the people who built the device are easiest to fool. Demonstrations, selected runs, trained operators, healthy volunteers and internal projections can all look like performance without being it. Integrity here means putting on the table everything that could make the result wrong: failed runs, repeats, samples too small to run, results produced on other equipment, and patients whose values did not fit. Nothing in the packet concerns this device, so the core of this hole is reasoning. The packet's one retracted record about measurement itself, E4, concerns assay problems and their impact on published pharmacokinetic evaluations; being retracted, it is a precedent of failure on measurement validity, but its abstract is cut off and it is about a drug assay, not capillary sampling, so it supports only the general warning.

    Rests on 1 retrieved record: E4
  3. minor'Comprehensive' is a name, not a test menu

    Testing single analytes outside the laboratory is not new: home blood glucose strips were in use by 1982 (E13), and a capillary prothrombin-time monitor showed acceptable agreement with one of its laboratory comparison methods in a 1995 multicentre study (E10). So the new part of the claim is the word 'comprehensive', and the word says nothing about which tests are run, from how much blood, or how well. Read with 'instead of venous blood draws', it promises the central laboratory's menu, which is the reading attacked here; if the company means a handful of tests, patients will still need venous draws and investors are hearing more than the device does. The menu has to be written down before anything else can be checked. The examples come from the packet, not the lens.

    Rests on 2 retrieved records: E10, E13
Claude ShannonWhich distinctions survive transmission, and which are lost?
  1. seriousA precise reading of an unrepresentative drop is still wrong

    The analyser can only report what is in the drop it is given. If the drop differs from the blood in the patient's veins, a perfect measurement passes that difference on faithfully; faithful transmission is not the same as truth, and this noise enters before the analyser does. E1 measured it directly for haemoglobin: repeat readings from single finger-prick drops differed by 10 g/l or more in 3 of 20 pairs collected by scientists and 7 of 20 collected by health visitors from the first drop, and in 1 of 20 (scientists) against 11 of 20 (health visitors) from the fourth drop. The error depended on who collected and which drop was used. That is one analyte, on another device, in healthy volunteers, and the abstract stops before its results on mixed drops, so it does not show the problem cannot be fixed. It does show that 'no clinically meaningful error' cannot be assumed for capillary drops, and at scale collection will rest with many people who are not laboratory scientists. The data come from the packet (E1, 1998), not the lens; the channel framing is an analogy, and the quantity to measure is drop-to-drop and collector-to-collector variation on this device.

    Rests on 1 retrieved record: E1
  2. seriousOne finger-prick has to carry the whole menu

    A venous tube holds enough blood to run a broad menu and still repeat a doubtful result, or dilute and rerun one that is out of range; that spare sample is the redundancy that protects a laboratory's results against noise. A few drops divided across chemistry, haematology and immunoassays leave each test a sliver and nothing in reserve, so errors cannot be caught by repeating, and the claim may ask the sample to carry more than it can. Many chemistry and immunoassay targets are plasma constituents (E5 calls plasma a rich source of protein and metabolite biomarkers), so the device will usually have to separate plasma from a tiny volume, or show that its whole-blood results match plasma results. E5, a 2013 review, says a significant part of plasma preparation was still done manually and off-chip, by centrifugation or filtration, warns that non-reproducible preparation can degrade the analytes of interest, and presents automated, reproducible miniaturised separation as a goal. That does not show this device cannot do it; it shows the step needs its own evidence. The plasma part depends on the packet, not the lens.

    Rests on 1 retrieved record: E5
Rachel CarsonWhere will the costs appear after the benefits have been counted?
  1. fatalScale moves the cost of wrong results onto patients, where it surfaces late

    On what is presented, the evidence for going to scale is the company's own description, and the first hole is still open. Scale is where a diagnostic's error rate stops being an anecdote: a small share of wrong results becomes a large number of wrong decisions. The costs are displaced and delayed. A falsely normal result is seldom rechecked, so in routine use nobody notices it; it returns later as a missed or late diagnosis, borne by a patient and a clinician who never saw the sample. A falsely abnormal one sets off repeat tests and treatment. Moving testing out of the central laboratory also risks removing the checks that catch errors there, such as repeat runs, quality control and trained staff, unless the device replaces each one. Patients asked to rely on these numbers have a right to know how well each test performs. The ecological framing is an analogy and has to be tested in this domain, which is what the test below does. Regulation lies beyond the lens and depends on the packet: its regulatory sources concern direct-to-consumer genetic tests, and the one with an abstract (E3) says only that medical-device and clinical-laboratory frameworks both bear on such tests and that stronger oversight has been called for.

    Rests on 1 retrieved record: E3

Killer testA blinded split-sample study on the analyser itself. In consecutive patients who need a venous draw anyway, so that abnormal values are included, one finger-prick is collected by the kind of staff who will use the device at scale and run on the full claimed menu, and a venous sample taken at the same visit goes to an accredited reference laboratory for the same tests. An independent party holds the key and logs every attempt, including failed, repeated and insufficient samples and the instrument that produced each result. Before unblinding, record for each test the allowable difference, set at how closely two accredited laboratory methods agree on the same venous samples, and the share of patients allowed to need a second sample. Analyse differences and changed clinical decisions, not correlation.

14 records retrieved before the review was written; 10 citations checked against them, 0 stripped.

What happened · 14 March 2018

It failed. The SEC charged the company and its leaders with fraud: the analyser could complete only a small number of tests, and most patient tests ran on modified commercial analysers made by others (14 March 2018).

Their verdicts differed: the standard AI said “Breaks”, Feynman “Has a serious hole”.

008How it was run, and how it will be scored.For the sceptical

This row is the machinery: how each run was held to its date, what a review does that a plain answer doesn’t, and how big a test it would take to settle which is better. Skip it if you only want the answers. Check it if you don’t trust them.

001 / Evidence stops at the date

Every search was limited to work published before the idea’s date. The filter was added to each request to OpenAlex on its way out, so the product’s own search code ran unchanged. Retraction flags are today’s, the one leak of hindsight in the record.

002 / Rules before results

The ideas, outcomes and four scoring rules were written down and fingerprinted before any review ran: verdict, top hole, killer test and clean citations. SHA-256 175c5d5a5ee6…

003 / A person checks the scores

Claude scored each review first. Rule scores reach this page only after a named person has checked every one, and that has not happened yet.

What a review adds to one answerThese runs

Model calls1 plain · 2 in a review

Records retrieved before the cutoffnone · 9–14 per idea

Searches per ideanone · 4 and a retraction pass

Citations checked against recordsnone · 35 checked, 0 stripped

Ways of reasoningone · three lenses, D = 2.75–2.94

Works cited75 from memory, unchecked · 35 to retrieved records

Problems raised5–6 · 5–7 ranked, on 6 typed claims

Held to a dateno · the evidence, yes; the model’s memory, no

In each line the plain answer comes first and the Feynman review second. It says nothing yet about which answer is closer to the truth. The lenses run on one model, so their errors are not independent, as the maths shows. D is the effective number of lenses the holes came from (the Hill–Shannon measure): 1 means one lens raised them all, and 3 means all three contributed equally. The lenses were then chosen from 23.

Our home page promises one comparison: Feynman against one strong model asked the same question, published either way. To mean anything it needs ideas whose endings the model cannot know, answers scored by people who can’t see which is which, and enough cases. Compared idea by idea, only the ideas where the two disagree carry information (that is McNemar’s test), so the number of cases matters.

001 / Endings it cannot know

Outcomes that arrived after the model’s training data ends, or reviews registered now and scored when the result lands. A probe first asks the plain model what happened, and any case it can answer is dropped.

002 / Five arms, one model

A plain answer; an expert prompt asking for what Feynman gives, once and five times over; the expert prompt given the records Feynman retrieved, which isolates what retrieval adds; and the full review. The design is fixed and fingerprinted.

003 / Scored blind, by people

Two named people score every answer against rules fingerprinted in advance, without knowing which arm wrote it. Their agreement is reported.

only Feynman’s review rightonly the plain answer rightboth agree, so the case tells the test nothing

33cases needed
40%that disagree
30%the gap

Two-sided α = 0.05 and 80% power. Sample size after Connor (1987), Biometrics 43(1) for the test of McNemar (1947), Psychometrika 12(2).

  1. Not shown / Foresight

    That Feynman foresees outcomes. The model knew these endings, and its searches say so.

    No
  2. Not shown / Better than plain

    That a review beats a plain answer. The plain answers above are real, but no person has scored either column, and the model knew how these cases ended. The comparison that could settle it is the one sized above.

    Not yet
  3. Not shown / The scores

    That the scores are right. They are Claude’s first pass until a named person checks them, and only then do they appear here.

    Unchecked
  4. Not shown / The product exactly

    That the runs match the product exactly. Each model call ran as a Claude Code subagent given the product’s prompts as task text, without the product’s effort settings or its fallback to another model when the first declines.

    Close