Answer in brief: Six claims were supported, five were stale or unsupported, and four were unverifiable. Confidence and specificity were poor proxies for truth.
What fails when persuasive market language is checked against primary sources? A useful answer has to be narrower than a product claim. This article tests a bounded workflow, publishes the scoring surface, and keeps consequential approval with a person. It does not turn a controlled result into personalized financial advice.
What we tested or analyzed
We asked OpenAI GPT-5.6 in a Codex editorial session to evaluate 15 fixed synthetic stock-analysis claims and then applied a primary-source/time-stamp rubric. No ticker was recommended and no trade was simulated.
The original asset is a fifteen-claim source and time-sensitivity audit. The complete machine-readable table is available as CSV. The evidence visual below summarizes the primary criterion; its values are also written in text and shown in the table, so the chart is not the only way to obtain the result.
The measured result
Six claims were supported, five were stale or unsupported, and four were unverifiable. Confidence and specificity were poor proxies for truth.
The row-level outcome distribution was stale: 2, supported: 6, unsupported: 3, unverifiable: 4. Those labels are deliberately more descriptive than one blended score. A partial, review, stale, exception, or unsupported row can carry a different operational risk from a plainly wrong row, so the CSV preserves the reason beside the disposition.
| Item | Outcome | Evidence or note |
|---|---|---|
| C01 | supported | filing fact |
| C02 | supported | dated price fact |
| C03 | supported | share-count fact |
| C04 | supported | risk disclosure |
| C05 | supported | dividend history |
| C06 | supported | reported revenue |
| C07 | stale | old margin |
| C08 | stale | old guidance |
| C09 | unsupported | market leadership |
| C10 | unsupported | competitive moat |
| C11 | unsupported | inevitable growth |
| C12 | unverifiable | private pipeline |
| C13 | unverifiable | future price |
| C14 | unverifiable | rumored partnership |
| C15 | unverifiable | guaranteed return |
Reading the evidence row by row
- C01 was recorded as supported. The evidence note is “filing fact”; the disposition remains visible so it cannot be averaged away.
- C02 was recorded as supported. The evidence note is “dated price fact”; the disposition remains visible so it cannot be averaged away.
- C03 was recorded as supported. The evidence note is “share-count fact”; the disposition remains visible so it cannot be averaged away.
- C04 was recorded as supported. The evidence note is “risk disclosure”; the disposition remains visible so it cannot be averaged away.
- C05 was recorded as supported. The evidence note is “dividend history”; the disposition remains visible so it cannot be averaged away.
- C06 was recorded as supported. The evidence note is “reported revenue”; the disposition remains visible so it cannot be averaged away.
- C07 was recorded as stale. The evidence note is “old margin”; the disposition remains visible so it cannot be averaged away.
- C08 was recorded as stale. The evidence note is “old guidance”; the disposition remains visible so it cannot be averaged away.
- C09 was recorded as unsupported. The evidence note is “market leadership”; the disposition remains visible so it cannot be averaged away.
- C10 was recorded as unsupported. The evidence note is “competitive moat”; the disposition remains visible so it cannot be averaged away.
- C11 was recorded as unsupported. The evidence note is “inevitable growth”; the disposition remains visible so it cannot be averaged away.
- C12 was recorded as unverifiable. The evidence note is “private pipeline”; the disposition remains visible so it cannot be averaged away.
- C13 was recorded as unverifiable. The evidence note is “future price”; the disposition remains visible so it cannot be averaged away.
- C14 was recorded as unverifiable. The evidence note is “rumored partnership”; the disposition remains visible so it cannot be averaged away.
- C15 was recorded as unverifiable. The evidence note is “guaranteed return”; the disposition remains visible so it cannot be averaged away.
The expected label or control was fixed before review. The visible note explains why the row received its disposition. The chart uses the published primary criterion, but the table is authoritative because it preserves exceptions that a single percentage would hide.
How to reproduce the check
- Download the CSV and read its labels, units, and synthetic/public-data notice before using it.
- Write the expected answers or decision rule before looking at a model response.
- Use the same bounded prompt and record the model or tool, access surface, and date.
- Preserve the raw response. Break prose into atomic claims rather than grading the tone of the whole answer.
- Recompute arithmetic with deterministic formulas and verify definitions against the linked primary sources.
- Record correct, partial, wrong, uncertain, and refused outcomes separately. Do not silently repair the model output before scoring it.
- Repeat material checks after a model, source, or workflow changes.
What the result means
The value of this result is diagnostic. It shows where a structured assistant can reduce search, formatting, or first-pass review work. It does not transfer responsibility for the underlying decision. A “pass” means the row met the published rule in this test, on this date, with these inputs.
The errors and open items matter more than a polished average. In money and business workflows, one missed assumption, stale fact, false match, or overconfident definition can dominate many correct low-risk rows. That is why the artifact keeps row-level outcomes and why a human reviews exceptions rather than receiving only a percentage.
Reproducibility also has limits. A reader can repeat the steps and inspect the same answer key, but a probabilistic model may not return identical wording. A useful rerun should therefore compare atomic claims, calculations, citations, and escalation decisions—not superficial phrasing.
Why this topic needs its own boundary
Market research becomes unreliable when dates disappear. A reported fact, a management forecast, a market price, and an analyst inference have different sources and time horizons. The workflow keeps those evidence types separate and rejects persuasive language as a substitute for a filing or time stamp.
None of the tests estimates intrinsic value or recommends a security. Their purpose is claim hygiene: finding where a statement came from, what period it describes, what could contradict it, and whether the statement is supportable at all.
A safer operating workflow
- Require a source and “as of” date for every market fact.
- Reject guarantees and hidden assumptions.
- Distinguish reported history from forecasts.
- Check registration and official filings.
- Consult a qualified professional for personal investment decisions.
How each control changes the decision
Control 1: Require a source and “as of” date for every market fact. For this test, that control answers the bounded question “What fails when persuasive market language is checked against primary sources?” without extending the result into an untested decision.
Control 2: Reject guarantees and hidden assumptions. For this test, that control answers the bounded question “What fails when persuasive market language is checked against primary sources?” without extending the result into an untested decision.
Control 3: Distinguish reported history from forecasts. For this test, that control answers the bounded question “What fails when persuasive market language is checked against primary sources?” without extending the result into an untested decision.
Control 4: Check registration and official filings. For this test, that control answers the bounded question “What fails when persuasive market language is checked against primary sources?” without extending the result into an untested decision.
Control 5: Consult a qualified professional for personal investment decisions. For this test, that control answers the bounded question “What fails when persuasive market language is checked against primary sources?” without extending the result into an untested decision.
Keep data collection, model preparation, deterministic validation, and approval as separate stages. Use the least sensitive input that can answer the question. If removing personal or confidential data makes the result ambiguous, route the case to an approved human process instead of restoring secrets to an unapproved tool.
Calculations need an independent formula; current facts need a current primary source; classifications need an “uncertain” route; and irreversible actions need explicit authorization outside the model. Logs should capture the version, prompt, source date, output, reviewer, correction, and final disposition without retaining unnecessary personal data.
Limitations and professional boundary
The set is small and designed to test claim hygiene, not market forecasting. Prices and filings change.
This publication provides general educational information. It does not know a reader’s finances, duties, jurisdiction, contracts, tax treatment, credit position, or risk tolerance. A qualified financial, accounting, tax, legal, lending, security, or other professional should review decisions with material consequences.
Primary sources
Verified 2026-07-26. Primary-source links can change; use the publication date and linked source to check for a newer version.
Bottom line
Six claims were supported, five were stale or unsupported, and four were unverifiable. Confidence and specificity were poor proxies for truth. The practical lesson is to make AI produce inspectable work inside a controlled process—not to make fluency the final control.