AI Accuracy on Financial Documents: 18 Models Tested on a 60-Page 10-Q

Introduction

Financial firms using Claude or ChatGPT to automate routine work kept hitting the same problem with 60–100 page PDFs: accuracy. When professionals asked questions about those documents, the tools would often “forget” or “make up” key figures that underpin any analysis an investor may be working on. A growing group of firms uses more specialized tools that try to close this gap, but their cost is often higher than firms feel is worthwhile. So we asked what a bare-bones system built on an off-the-shelf LLM would need to reliably extract specific figures, and do limited analysis, from documents that size. The same approach could apply to any industry that routinely pulls specific numbers out of long, unformatted documents.

We considered consumer tools like NotebookLM, Claude, and ChatGPT, but wanted to avoid their built-in scaffolding so the lessons would apply to any LLM-powered tool. We started in a Claude Code session, with two goals: a proof of concept that extracts specific figures accurately, and a way to test which variables affect that accuracy.

First attempts in Claude Code

Our first attempts were conversational: describe the problem, rerun, describe it again. In one session, getting the model to return earnings per share as both a value and its calculation took eight exchanges, and the fix still didn’t hold when we tried again in the same session.

When we moved away from conversational prompting and toward carefully worded instructions, accurate results started to become more consistent. One of the specific instructions we first arrived at was using a bound of the expected value. Those bounds acted as guideposts, telling the model when to pause and verify its output:

“Combined Ratio should equal Loss Ratio + Expense Ratio, within 1–2 points”

After many exchanges learning which instructions lead to success most often, we had prompts that worked well enough and we wanted to test which combination of prompt, model, and text-extraction method produced the most accurate results.

Setup: FinLab, inference, and the two prompts

We built a small web application, FinLab, to store prompts, documents, and our human-identified “correct” values. In FinLab, we could run models using these constants while varying model, prompt, and extraction method.

Figure 1 - FinLab’s run form. The document, technique, model, and prompt shown are placeholders, not a study configuration. Submitting shows a cost estimate before anything runs.

LLM inference ran through DigitalOcean’s serverless platform, which let us test many models through one provider instead of setting up individual accounts with Anthropic, OpenAI, Meta, and others. Furthermore, DigitalOcean states that its serverless inference runs under zero data retention for certain models (see the data-privacy link in the appendix). Of course, a firm with SEC, FINRA, or HIPAA obligations should review and validate that policy as part of its own vendor due diligence before sending client data to any AI service.

Once the infrastructure was set up, we tested 18 models, three text-extraction techniques, and two prompts over four rounds, all on a public Travelers Companies 10-Q. We wrote two prompts. The general prompt is meant to work across companies and document types. It is what a professional would reach for with a new client. The company-specific prompt is tuned to one company they work with repeatedly.

Prompt Development

Conversational prompting had a second cost: whatever the model had learned in one session was lost in the next. A professional on a deadline could spend their time getting the model back to where it was for the last memo, before getting any answer for this one.

A maintained CLAUDE.md file fixed that. Any persistent system instruction works the same way, including project instructions in Claude, ChatGPT, or Copilot. Tasks became repeatable and past mistakes stopped resurfacing. An example from ours:

“Ultimate Goal: Handle private market documents that arrive via email in various formats (PDFs, Word documents, Excel sheets), not just structured SEC EDGAR filings.”

Extraction itself was harder than expected. Early prompts returned a number near the keyword we wanted, such as EPS, but not the correct value. Adding an expected range fixed most of these instances, and we kept using ranges as a signal for the model to double-check before answering:

“Number of Shares Outstanding” is often a decimal in millions (e.g. “212.6”) — don’t confuse with a raw share count.

We also wrote explicit rules. Some models, especially the newest, answered with questions instead of data until told not to:

“7. Output must be valid JSON only — a single JSON object, with no explanation, no prose, and no text before or after the JSON. Do not wrap the JSON in markdown code fences or any other formatting; output raw JSON starting with “{“ and ending with “}”.

During prompt development, several models, especially the newest ones, needed this rule to return only raw data, while GPT-5.4 Mini returned raw data with or without it.

Early rules were vague; later ones were precise. An early version of our general prompt simply said, “Return your answer as a single JSON object with metric names as keys and values as strings.” By v10, the rule for calculated metrics read:

“9. For any calculated metric, you must provide both of the following — never only one: (a) a primary key holding the specific, concrete final answer — never blank, never a placeholder, and never the arithmetic expression itself; and (b) a separate key showing the arithmetic that produced or checks that answer…”

Ambiguous language and assumed context caused most errors. We learned to state everything explicitly, down to the exact JSON format because models would not infer it.

Non-determinism

Models also gave different answers to identical inputs: same document, prompt, and technique. We expected non-determinism, so we report ranges over repeated runs rather than single scores. For professionals the lesson is practical: give the model reference points (if Company X earned $100 million last quarter, say so, and a $6 billion answer this quarter will stand out), run important workflows more than once, and check the work when results differ.

Claude Fable 5.1, same inputs, two runs:

Run 1

“1,711 / 212.6 = 8.048 (vs reported basic EPS 7.89, which uses weighted average basic shares 215.2 and net income available to common of 1,699: 1,699 / 215.2 = 7.895)”

Run 2

“1,711 / 212.6 = 8.048 vs reported basic EPS 7.89 (difference due to basic weighted average shares of 215.2 million and $12 million allocated to participating awards: 1,699 / 215.2 = 7.895, matches)”

Both runs reached the same final answer, but the reasoning differed. Part of the difference came from our own prompt: our EPS check divided net income by shares outstanding, but basic EPS uses the weighted-average share count. Both runs noticed and reached the reported figure anyway; a less careful run could have trusted our flawed check instead.

Method

To grade a run, we scored the 20 metrics that both the general and specific prompts asked for, weighing each answer at 0.05, so a perfect run scored 1.0. The company-specific prompt also asked the model to calculate Total Debt from short- and long-term debt. That calculation helped us check the model’s work, but we didn’t count it in the score.

Figure 2 - FinLab’s per-metric scoring for one perfect run. FinLab scores each metric 0 or 1; the post reports the mean across the 20 metrics, which is the same as weighting each at 0.05.

Beyond accuracy we tracked cost per run, prompt sensitivity (how much results depend on exact wording, which sets the learning curve), timeout rate (a model that rarely completes isn’t practical), and response time (a model as slow as doing it by hand adds little).

We tested in four rounds. In rounds 1 and 2, each model ran once on all six combinations of extraction technique (A, B, or C) and prompt (general or company-specific), for up to 12 scored runs per model; the appendix shows round 2 only. In round 3, each model ran three times on the best-scoring general and company-specific combinations from round 2: Technique A with the general prompt (average score 0.84) and Technique B with the company-specific prompt (0.98). We also reran GPT-5.4 Mini several times on the general prompt to investigate low scores; those extra runs are listed but not scored. In round 4, we revised both prompts and reran 13 of the models three more times on those same two combinations.

The ‘final round’ results in the Results section and in Table 3 come from round 4. Where a run failed we retried, and in two cases a fourth run completed. We report the first three completed runs for every model and note extra or failed attempts in the appendix.

Models released mid-project (GLM-5.3 Flash, Nemotron 3 Ultra, GPT-6 Sol, Astra and Luna, Claude Sonnet 5.5, Claude Opus 5.5, and Claude Fable 5.1) joined in round 3, so they ran only on the two carried-forward combinations and weren’t part of selecting them. Qwen 3.5, Llama 4 Maverick, GLM-5.3 Flash, and Nemotron 3 Ultra don’t support PDF uploads, so they never ran Technique A; in rounds 3 and 4 they ran only on Technique B with the company-specific prompt.

The two prompts differed mainly in how much company context they gave the model.

General: “Note: Most insurance ratios are expressed as a percentage (%). Look for tables titled “Results of Operations” or “Segment Results” and read the paragraph above each % value to confirm what ratio it refers to.”

Specific: “Combined Ratio should equal Loss Ratio + Expense Ratio, within 1–2 points. Travelers typically runs Combined Ratio 87–95%, Loss Ratio 58–65%, Expense Ratio 28–31% for a quarter.”

Results

The clearest evidence that the prompt mattered came from the GPT-6 models. In round 3, GPT-6 Sol and Astra scored 0.1 on every general-prompt run, and Luna scored as low as 0. Between round 3 and round 4 we revised the general prompt: the earlier version had asked for the most recent annual figure on a quarterly filing, so we corrected that, and we also added a rule to round every number to three decimal places. With the revised prompt, all three GPT-6 models scored 0.95–1.0.

Figure 3 - Mean score by model, Technique A with the general prompt, round 4. First three completed runs per model. Shaded rows had failed or extra attempts. Four models could not run Technique A.

Figure 4 - Mean score by model, Technique B with the company-specific prompt, round 4. First three completed runs per model. Shaded rows had failed or extra attempts.

The biggest surprise came from the cheapest models we tested (Figures 5 and 6). GPT-5.4 Mini cost about $0.06 per run on the general prompt and $0.04 on the specific one, with average final-round response times of 6.8 and 10.0 seconds. In the final round, it scored 1.0 on four of six runs and 0.95 on the other two. GPT-5.4 Nano cost about $0.02 and $0.01 per run, averaged 7.7 and 11.3 seconds, and scored 1.0 on every specific-prompt run. For comparison, a general-prompt run cost about $1.72 on Claude Fable 5.1 and $1.46 on GPT-6 Astra.

The mid-priced GPT-5.6 models were less predictable. In round 2, GPT-5.6 Sol scored 1.0 on every company-specific run but only 0.1 on the general prompt with Techniques B and C. GPT-5.6 Terra (about $0.37 per general-prompt run) was inconsistent even with nothing changed: in round 3, three back-to-back runs with the same prompt and technique scored 0.95, 0.1, and 1.0.

Figure 5 - Score per run, Technique A with the general prompt, round 4. Each dot is one of the first three completed runs. Nano’s fourth run (0.95) is excluded under the first-three rule.

Figure 6 - Score per run, Technique B with the company-specific prompt, round 4. Each dot is one of the first three completed runs. Qwen had one failed attempt and one extra completed run (1.0); GLM completed one of five attempts.

The Claude models reinforced the same lesson. Haiku 4.5 was the standout, scoring a perfect 1.0 on all six final-round runs despite being a budget model, at about $0.15 per general-prompt run and $0.05 per specific-prompt run, with average response times of 6.5 and 12.6 seconds (Figure 7). Sonnet 5.5 was nearly as strong, with five perfect scores and a single 0.95. Opus 5.5 ran 0.95–1.0, perfect on every specific-prompt run but dipping twice on the general prompt. Fable 5.1, the model we were most excited about, was consistent (0.95–1.0) but hit only two perfect scores, underperforming for its cost.

Among the GPT models, GPT-6 Astra led with five perfect runs, while GPT-5.4 Nano was perfect on the specific prompt but fell to 0.75 once on the general one. The company-specific prompt revision had a smaller but consistent effect: most models moved from 0.85 to 0.95 on v9 to 0.95 or 1.0 on v10, and GPT-6 Astra from 0.2 to 1.0.

In the final round we ran the open models only on the specific prompt: Nemotron 3 Ultra scored 1.0 on all three runs, Qwen 3.5 scored 1.0 on two of three completed runs and 0.9 on the other, with each run taking 25 to 29 minutes. Llama 4 Maverick held at 0.9, and GLM-5.3 Flash completed only one of five attempts (0.95).

Recommendations

No single model won every category across techniques and prompts. Claude Haiku 4.5 stood out overall, combining low cost, top accuracy, near-perfect consistency, and fast responses. GPT-5.4 Nano is a solid budget option when paired with a specific prompt, while GPT-6 Astra and Claude Sonnet 5.5 also delivered top accuracy.

For most organizations, we recommend Claude Haiku 4.5. GPT-5.4 Mini is a strong alternative: it scored perfectly on the specific prompt, was easy to work with, and cost about $0.04–$0.06 per run. Before choosing it, check whether it’s offered in the AI tools your firm already uses.

More broadly, firms should test cheaper models on tasks like extraction and limited analysis before defaulting to premium ones. These results have a short shelf life. Eight of the 18 models we tested were released while the project was running, and prices changed during it. The specific ranking will be out of date within months. What should last is the method: a small set of documents with known correct values, a fixed scoring rule, three runs per configuration, and a cheap model tested before an expensive one. Most firms should keep a version of that and rerun it when a new model or price arrives.

Future Research

Future research could focus on checking compliance for post-closing earn-outs and purchase price adjustments. Moving from extracting data out of one document to cross-checking two would be the next step: can a model correctly extract a contract’s formula and apply it to actual financials? Earn-out disputes are well documented and costly, so catching potential points of disagreement early could save real time and money. This work could also test verification-focused models, which check answers rather than generate text. Paired with an LLM that handles extraction and calculation, such a model could act as a fast, low-cost check on the results. Because earn-out work involves confidential deal terms, any vendor’s data-retention commitments should be confirmed in a signed agreement before real contract data is used.

Appendix

You are a financial analyst extracting key metrics from the attached document — a quarterly report (Form 10-Q) for Travelers Companies. Use only the document provided. Do not rely on prior knowledge of Travelers' financials or any figures you may recall from training — every value must come from the attached document.

**Financial identities to check before answering.** Combined Ratio should equal Loss Ratio + Expense Ratio, within 1–2 points. Travelers typically runs Combined Ratio 87–95%, Loss Ratio 58–65%, Expense Ratio 28–31% for a quarter. Consolidated Net Premiums Written for a single quarter should exceed $5 billion — if you get something much smaller, you've landed on a segment subtotal (Business Insurance, Personal Insurance, or Bond & Specialty), not the consolidated Total row.

**Current period only.** Every table in this filing has a current-quarter column and a prior-year comparative column next to it. Extract only the current quarter. Balance sheet items (debt, cash, assets, liabilities, equity) show a current period-end column next to a prior fiscal year-end column — take the current one. A debt or cash figure that matches a suspiciously familiar prior-period number is a sign you read the wrong column.

If a number feels off, look again. Return "N/A" rather than guessing.

### Rules

1. Return the most recent quarter only — ignore prior-year and prior-period comparative columns in every table.
2. Include the unit in every value (millions, billions, %, etc.).
3. If a value is not found, return "N/A" — do not make up or guess values.
4. Do not add metrics or keys beyond what is listed below.
5. Do not use parentheses to indicate negative values — use a leading minus sign instead.
6. Keep every value terse: the number and unit only, with no additional commentary.
7. Output must be valid JSON only — a single JSON object, with no explanation, no prose, and no text before or after the JSON. Do not wrap the JSON in markdown code fences or any other formatting; output raw JSON starting with "{" and ending with "}".
8. "Earnings Per Share (EPS)" means basic EPS, not diluted EPS.
9. For any calculated metric, you must provide both of the following — never only one: (a) a primary key holding the specific, concrete final answer — never blank, never a placeholder, and never the arithmetic expression itself; and (b) a separate key showing the arithmetic that produced or checks that answer (e.g. `"Combined Ratio (calculation)"`, `"Earnings Per Share (EPS) (calculation)"`, `"Total Debt (calculation)"`). If the schema doesn't list a plain key for one of the numbered metrics below, add one — do not use the calculation key as the only place that metric's answer lives. (`"Balance Sheet Identity Check"` is the one exception: it's a standalone cross-check between metrics that already have their own primary keys, not a separately listed metric itself, so it doesn't need a plain sibling.)
10. Consolidated totals only — Travelers reports three segments; always take the Total row.
11. If the document says "in millions except per share," scale bare table numbers × 1,000,000.
12. Round every numeric value to the nearest three decimal places (e.g. "$10.33842 billion" → "$10.338 billion").

### Travelers-Specific Notes

- "Net Premiums Written" appears in a "Consolidated Written Premiums" table. Use the Total row, not the Business Insurance or Personal Insurance segment rows — this is the single most common extraction error on this filing.
- "Loss Reserves" is "Claims and claim adjustment expense reserves" on the balance sheet and in Note 7.
- "Reserve Development" is "net favorable (unfavorable) prior year reserve development" in the MD&A reserves discussion.
- "Total Expenses" is "Total claims and expenses." "Operating Costs" is "General and administrative expenses."
- "Catastrophe Losses" is "Catastrophe losses" or "Net catastrophe losses" in MD&A/Segment Information.
- "Cash" includes restricted cash — line is "Cash and restricted cash at end of period."
- "Short-term Debt" and "Long-term Debt" are both on the balance sheet — confirm you're in the current period-end column, not the prior fiscal year-end comparative.
- "Number of Shares Outstanding" is often a decimal in millions (e.g. "212.6") — don't confuse with a raw share count.

### Metrics to Extract

1. **Net Income** — Consolidated Statement of Income, current quarter; "Net income" attributable to Travelers
2. **Earnings Per Share (EPS)** — basic EPS, not diluted, current quarter
3. **Combined Ratio** — MD&A Results of Operations table
4. **Loss Ratio** — same table; "Loss and loss adjustment expense ratio"
5. **Expense Ratio** — same table; "Underwriting expense ratio"
6. **Net Premiums Written** — Consolidated Written Premiums table, Total row; expect >$5B for a quarter
7. **Net Premiums Earned** — Consolidated Statement of Income; "Premiums" or "Net earned premiums"
8. **Catastrophe Losses** — MD&A or Segment Information note, net of reinsurance
9. **Loss Reserves** — balance sheet or Note 7; "Claims and claim adjustment expense reserves"
10. **Reserve Development** — MD&A reserves discussion; net favorable/unfavorable
11. **Total Revenue** — Consolidated Statement of Income; "Total revenues"
12. **Total Expenses** — "Total claims and expenses"
13. **Operating Costs** — "General and administrative expenses"
14. **Cash** — Statement of Cash Flows, "Cash and restricted cash at end of period"
15. **Short-term Debt** — balance sheet, current period-end column
16. **Long-term Debt** — balance sheet, current period-end column
17. **Total Debt** — Short-term Debt plus Long-term Debt (calculate this from the two components)
18. **Total Assets** — balance sheet, current period-end column
19. **Total Liabilities** — balance sheet, current period-end column
20. **Shareholders Equity** — balance sheet; "Shareholders' equity"
21. **Number of Shares Outstanding** — cover page or equity note; decimal millions

### Return Format

**Your response must be valid JSON only.** Output a single JSON object — no explanation, no prose, no text before or after the JSON. Use `"N/A"` (as a string) for any metric you cannot find with confidence.

For every calculation key, show your arithmetic using **only the values you just extracted** — this lets a reviewer check your work without re-reading the filing. If an identity doesn't hold within tolerance, re-check the two inputs; if it still doesn't hold, answer anyway but flag the mismatch in the calc string (e.g. `"58.7 + 28.6 = 87.3 (matches)"` or `"... vs reported 91.0 (MISMATCH — re-check inputs)"`). The paired primary key (e.g. `"Combined Ratio"`, `"Earnings Per Share (EPS)"`, `"Total Debt"`) must always be the final number only — never the arithmetic expression; that belongs exclusively in the calculation key. (`"Balance Sheet Identity Check"` has no primary-key pair — it's a standalone cross-check, not a separately listed metric.)

```json
{
  "Company": "exact legal name from the filing cover page",
  "Period End": "quarter end date, e.g. 2026-03-31",
  "Net Income": "value with units",
  "Earnings Per Share (EPS)": "$X.XX",
  "Combined Ratio": "XX.X%",
  "Loss Ratio": "XX.X%",
  "Expense Ratio": "XX.X%",
  "Combined Ratio (calculation)": "Loss Ratio + Expense Ratio = X (compare to Combined Ratio)",
  "Net Premiums Written": "value with units",
  "Net Premiums Earned": "value with units",
  "Catastrophe Losses": "value with units",
  "Loss Reserves": "value with units",
  "Reserve Development": "value with units",
  "Total Revenue": "value with units",
  "Total Expenses": "value with units",
  "Operating Costs": "value with units",
  "Cash": "value with units",
  "Short-term Debt": "value with units",
  "Long-term Debt": "value with units",
  "Total Debt": "value with units",
  "Total Debt (calculation)": "Short-term Debt + Long-term Debt = X",
  "Total Assets": "value with units",
  "Total Liabilities": "value with units",
  "Shareholders Equity": "value with units",
  "Balance Sheet Identity Check": "Total Liabilities + Shareholders Equity = X (compare to Total Assets)",
  "Number of Shares Outstanding": "value in millions",
  "Earnings Per Share (EPS) (calculation)": "Net Income / Number of Shares Outstanding = X (compare to Earnings Per Share (EPS))"
}
```

# General Extraction Prompt

This is the base prompt template for Claude-based extraction runs.
Variables in `{curly braces}` are filled at runtime from the bucket config and document.

---

## Prompt Template

```
You are a financial analyst extracting key metrics from the attached document — a
financial filing for {company} in the {bucket['label']} sector. Use only the document
provided — do not rely on prior knowledge of this company's financials or any figures
you may recall from training.

If a number feels off, look again. If you genuinely cannot find a value with
confidence, return "N/A" — that is more useful to an analyst than a plausible-looking
wrong number.

RULES:
1. Return the most recent figure only.
2. Include the unit in every value (millions, billions, %, etc.).
3. If a value is not found, return "N/A" — do not make up or guess values.
4. Do not add metrics or keys beyond what is listed below.
5. Do not use parentheses to indicate negative values — use a leading minus sign instead.
6. Keep every value terse: the number and unit only, with no additional commentary.
7. Output must be valid JSON only — a single JSON object, with no explanation, no prose,
   and no text before or after the JSON. Do not wrap the JSON in markdown code fences or
   any other formatting; output raw JSON starting with "{" and ending with "}".
8. "Earnings Per Share (EPS)" means basic EPS, not diluted EPS. If the filing reports
   both, use the basic figure.
9. Round every numeric value to the nearest three decimal places (e.g. "$10.33842 billion" → "$10.338 billion").

METRICS TO EXTRACT:
{metrics_list}

{synonyms_block}SEARCH HINTS — focus on these sections first:
{', '.join(bucket['search_hints'])}
Note: Most insurance ratios are expressed as a percentage (%). Look for tables titled
"Results of Operations" or "Segment Results" and read the paragraph above each % value
to confirm what ratio it refers to.

Example format:
{
  "Net Income": "$1.2 billion",
  "Earnings Per Share (EPS)": "$4.56",
  "Medical Care Ratio": "84.2%"
}

DOCUMENT TEXT (may be truncated):
{text}
```

---

## Example: `{metrics_list}` rendered for the `pc` bucket

```
- Net Income
- Earnings Per Share (EPS)
- Combined Ratio
- Loss Ratio
- Expense Ratio
- Net Premiums Written
- Net Premiums Earned
- Catastrophe Losses
- Loss Reserves
- Reserve Development
- Total Revenue
- Total Expenses
- Operating Costs
- Cash
- Short-term Debt
- Long-term Debt
- Total Assets
- Total Liabilities
- Shareholders Equity
- Number of Shares Outstanding
```

This is `bucket["extract"]` from `metrics.py` for `pc`, one item per line. The `health` and
`specialty` buckets render the same way from their own `extract` lists.

---

## Runtime Variables

| Variable | Source | Description |
|---|---|---|
| `company` | CLI arg / EDGAR metadata | Company name injected into the persona line |
| `bucket['label']` | bucket config | Human-readable sector label (e.g. "Property & Casualty") |
| `metrics_list` | bucket config (extract list) | Bullet list of metrics to extract |
| `synonyms_block` | bucket config (synonyms list) | Optional block of alternate field names; empty string if not defined |
| `bucket['search_hints']` | bucket config | List of document sections to prioritize |
| `text` | document fetch step | Full document text, capped at 1,200,000 characters |