# How StatementBench benchmarks are run

> The rules every StatementBench benchmark follows, the method of the bank-statement converter benchmark (the plan as published before measuring, and the pilot round run on public specimen statements in October 2026), measured results for the free converter on 17 published sample statements, and how the free tools and pricing data are tested.

By StatementBench. Updated 7 October 2026. Canonical URL: https://statementbench.com/methodology/

## Rules for every benchmark

1. **Measured, not estimated.** A number appears on this site only after it has been measured with the method published here. No results are inferred from vendor marketing or extrapolated from small samples.
2. **Data and method are published.** Each benchmark publishes its test documents, or redacted versions when the originals contain personal data, along with the expected output for each document, the scoring code, and the raw output of every tool tested.
3. **Same conditions for every tool.** Every tool runs on the same documents, within the same week, with documented settings. Settings are the defaults unless a setting is documented and applied to every tool.
4. **Conflicts are disclosed.** StatementBench builds a free converter that competes with the paid tools it tests, and may earn affiliate commissions from some of them. Both facts are stated on every benchmark page. The free converter is scored by the same harness, and its results are published even when it loses.
5. **Vendors can respond, not edit.** Before publication, each vendor gets its own results and may reply. Replies are published alongside the results. Results are not changed unless the vendor shows an error in the method, and then the fix and its date are recorded.
6. **Results are dated.** Tools change. Every result carries the date it was measured and the tool version where one is visible, and benchmarks are re-run when vendors ship major changes.

## Converter accuracy benchmark

**Status: in progress. A pilot round was run on 7 October 2026. Its results will be published after each tested vendor has seen its own results and had seven days to reply (rule 5).** The plan below was published on 6 October, before any converter was measured, and is kept as written so the method can't be bent to fit the results. The pilot round is smaller than the plan and differs from it in the ways listed under [Pilot round](#pilot-round-october-2026); each difference is also stated on the results page.

### Question

When you give a converter a bank statement, how often does it give you back every transaction, with the right date, amount and sign, without manual correction?

### Test set (planned)

- **Real statements, used with the account holder's consent**, redacted of names, addresses and account numbers. Redaction leaves amounts, dates, descriptions and layout intact.
- **Synthetic statements** produced from known ledgers in the layouts real banks use, so that edge cases can be covered on purpose: wrapped descriptions, page breaks inside a day, separate debit and credit columns, balances printed once a day, year boundaries, and statements with only opening and closing balances.
- **Scanned versions** of a subset, printed and scanned at fixed resolutions and also photographed, to measure tools that offer OCR. The free StatementBench tool doesn't do OCR and will be reported as unable to process these, not scored as zero on accuracy.
- **Target size:** at least 50 statements and 2,000 transactions from at least 10 banks before the first results are published. The final counts and the bank list will be published with the results.
- **Ground truth:** each statement's expected transactions are entered by hand and must reconcile with the statement's own balances before they are used.

### Tools (planned)

Up to eight converters: dedicated bank-statement converters such as DocuClipper, general document parsers, cloud OCR and extraction services, and the free StatementBench tool. The final list will be published with the reasons each tool was included.

### Scoring (planned)

- **Transaction recall and precision.** An extracted row matches a true row when the date and the amount (to the cent, with sign) are equal and the description is substantially the same. Missing rows count against recall. Invented or duplicated rows count against precision.
- **Field accuracy.** For matched rows: dates exactly right, amounts exactly right, signs right, and description character error rate.
- **Reconciliation pass rate.** The share of statements where the tool's output, with no manual edits, reproduces the statement's own opening and closing balances. This is the measure that matters most to a bookkeeper.
- **Effort and cost.** Time from upload to usable file, and the price to process the test set at each tool's public pricing on the test date.
- **Failure modes.** Every failed statement is classified (wrong year, merged rows, sign flipped, row missed, header taken for a transaction, and so on), so you can see how a tool fails as well as how often.

### What will not be claimed

The results will describe these documents and tools on the test dates. They will not claim a tool is "the most accurate" on statements unlike the test set.

### Pilot round (October 2026)

The pilot uses the public specimen statements described [below](#sample-set), because they can be uploaded anywhere without exposing anyone's data. It differs from the plan in these ways:

- **Test set.** 17 published specimen statements, 16 of them with a text layer, instead of 50 consented and synthetic statements. Eleven have hand-checked ground truth (174 rows): the five used while fixing the free converter, and six more checked on 7 October for this round. The plan's target set is still the goal before any result is called final.
- **How converters are run.** Before any use, each converter's terms of service are read and recorded. A converter whose terms forbid benchmarking, publishing results, competitive analysis or automated access is not tested. The others are used only through their free tier on their normal web page, one file at a time at a human pace, with default settings. No account is created, no payment card is entered, and no captcha, quota, rate limit or paywall is worked around: where a tool stops us, we stop and record it. Only the public specimens are uploaded.
- **Scoring.** Every converter's downloaded file, StatementBench's included, goes through the same code: a normalizer that finds the date, description, amount (or debit and credit) and balance columns by their header words, and the scorer used for the free converter. A row is correct when its date and signed amount match a ground-truth row to the cent; unlike the plan, descriptions are not required to match (description matches are reported separately). Two further rules: a file that uses the opposite sign convention throughout (card charges as negatives, say) is scored with its signs flipped, and that is reported; a date exported without a year counts as wrong in the main score, and a second score that ignores the year is reported beside it.
- **Reconciliation** is checked only on the four hand-checked samples whose own rows reach their printed opening and closing balances; on the others, even a perfect extraction cannot reconcile.
- **Not measured yet:** time from upload to file, the cost of processing the test set, and a full failure classification (the results page describes each failure in words). Vendor replies (rule 5) are requested before the results page goes live.

## Accuracy check: the free converter on published sample statements

**Measured 7 October 2026.** This is a check of StatementBench's own free converter, not a comparison with other converters; that is the separate benchmark above, whose pilot results are published once vendors have had their seven days.

### Sample set

Banks, credit unions, government agencies and educators publish sample statements to teach people how to read one. Every account holder in them is fictional. We collected 17 such PDFs that are public and need no login. We used no real person's statement and nothing leaked or uploaded by users.

- **By account type:** 10 checking statements (five of them also include savings, loan or credit line sections), 1 savings statement, 6 credit card statements.
- **By publisher:** 3 banks, 5 credit unions, 2 specimens from a credit union service organization, 3 government agencies, 3 schools or universities, 1 non-profit.
- **Text layer:** in 16 the statement has a text layer. In 1 the statement pages are images, like a scan.
- **Left out after review:** a public agency's real card statement posted on its website (a real account, not a sample), two worksheets marked for members or subscribers only, a software vendor's demo statement, a template with placeholder amounts only, copies of samples already included, documents with no transaction table (bank letter templates, an auto loan guide), and two samples whose servers refused scripted downloads (so the set could not be re-created).

These are teaching documents, not statements as banks deliver them. Some are annotated guides with callouts printed over and beside the statement, and many contain mistakes of their own. Checked by hand: seven have arithmetic errors (a printed balance that does not follow from the row above, or a summary that disagrees with the rows), one prints an amount on the wrong line, and one shows only the first page of the statement. Where a sample's own figures disagree, "needs review" is the correct result.

### Method

1. Each PDF is read with the same pdf.js version the browser tool uses (6.4.299, in its build for Node) and the same code that turns pdf.js output into positioned text. The converter runs with default settings: no year, date order or number format set by hand.
2. Each result is classed by the tool's own verdict: all rows reconcile; some rows or totals need review; rows extracted but nothing printed to check them against; or no rows found.
3. **Hand-checked ground truth for 11 samples (174 rows).** Five (137 rows) were made while fixing the converter; six more (37 rows) were added on 7 October for the converter benchmark. Every transaction line was typed from the document: the rendered page and its text, never the tool's output. Each sample's rows were checked against its printed balances before use; where the document disagrees with itself, its printed figures were kept. A row counts as correct when its date and its signed amount match to the cent (either the transaction or the posting date may match; a year printed as "20XX" is compared by month and day). Rows of 0.00 are left out. Card amounts are signed as the change in what you owe: charges positive, payments and credits negative.
4. **Precision** is correct rows divided by rows the tool extracted. **Recall** is correct rows divided by rows in the ground truth. **Amount accuracy** is, among the transactions the tool found, the share with exactly the right signed amount.

**The five hand-checked samples were also used to find and fix the failures**, so the scores after the fixes measure the fixes on documents the converter was adjusted against. They are not an estimate of accuracy on statements it has not seen. The six samples added later have no such tuning against their ground truth, but their layouts were among the 17 that shaped the fixes, so they are not unseen either. Each fix was also written as a test on a made-up statement with the same layout, so it is checked on every change.

### Results

| Outcome | Before fixes (code of 6 Oct 2026) | After fixes (7 Oct 2026) |
|---|---|---|
| All rows reconcile | 0 | 3 |
| Rows or totals need review | 11 | 9 |
| Rows extracted, nothing printed to check them against | 0 | 2 |
| No rows found | 6 | 3 |
| **Total** | **17** | **17** |

| Hand-checked subset (5 samples) | Before fixes | After fixes |
|---|---|---|
| Rows in the ground truth | 137 | 137 |
| Rows extracted (excluding 0.00) | 73 | 137 |
| Correct rows | 49 | 137 |
| Precision | 0.67 | 1.00 |
| Recall | 0.36 | 1.00 |
| Amount accuracy | 1.00 (49 of 49) | 1.00 (137 of 137) |

On the six samples hand-checked on 7 October (37 rows), the converter as fixed finds 33 rows correctly out of 34 extracted: precision 0.97, recall 0.89. It misses two Commerce Bank rows (deposit and ATM lines print the date after the description) and, on the City of Saint Paul sample, combines the Jan 19 paycheck with the check amount the sample prints on the same line. Across all eleven: 170 of 174 rows, precision 0.99, recall 0.98.

After the fixes, descriptions match the document exactly on 132 of the 137 rows. Of the other five, three also carry the reference number column, one a callout label printed inside the table, and one, a loan payment whose description is printed on the line above its date, shows the loan's other figures instead. Reading and checking took under half a second per PDF on a laptop.

Of the nine "needs review" results after the fixes, six point at mistakes or gaps in the sample itself (Carson Bank, Associated Credit Union, Scott Credit Union, CapEd, Boston Architectural College, NCUA). One points at interest the sample lists without dates (CU*Answers Black Bars). Two are failures of the converter (CU*Answers Two-Column, Commerce Bank). Carson Bank's result has a second cause: the converter cannot split its credit line payment, printed over two lines, into principal and interest.

| Sample (publisher's page) | Type | Rows | Result after fixes | Hand-checked | What happened |
|---|---|---|---|---|---|
| [Carson Bank](https://www.carsonbank.com/disclosure/) | Checking and credit line | 49 | Needs review | 48 of 48 correct | Plus one 0.00 line from a two-line loan record. The sample's summary box does not match its transaction table. |
| [Bank of America](https://www.bankofamerica.com/content/documents/CG2/How%20To%20Read/en_US/HTR-043235.pdf) (PDF) | Checking, annotated brochure | 8 | Nothing to check against | No | Statement excerpts are stacked images whose text layers overlap. Not scored. |
| [Associated Credit Union](https://www.acuonline.org/home/fiFiles/static/documents/how_to_read_your_statement.pdf) (PDF) | Savings, checking, credit line | 4 | Needs review | No | One row flagged: the sample's draft is $0.30 off its own printed balance. |
| [Scott Credit Union](https://web.archive.org/web/20250905141259/https://www.scu.org/statements/) (archived page) | Checking | 3 | Needs review | 3 of 3 correct | The sample's balance after a $1,306.50 draft does not follow from the row above. |
| [City of Saint Paul](https://www.stpaul.gov/departments/parks-and-recreation/right-track/youth/youth-resources) | Checking | 6 | All reconcile (totals only) | 5 of 7 correct | The sample prints one check's amount on another row's line, so one output row combines two transactions. A totals-only check cannot catch that. |
| [Consumer Financial Protection Bureau](https://www.consumerfinance.gov/consumer-tools/educator-tools/youth-financial-education/teach/activities/analyzing-credit-card-statements/) | Credit card | 0 | No rows | Correct (no rows) | Correct: the handout has no transaction list. |
| [University of Georgia](https://www.fcs.uga.edu/docs/credit_ho_5.pdf) (PDF; Federal Reserve sample) | Credit card | 0 | No rows | No | The statement pages are images. The converter reports two pages with no text layer. |
| [Randolph-Brooks FCU](https://www.rbfcu.org/learn/its-a-money-thing/using-your-credit-card) | Credit card | 9 | Nothing to check against | 9 of 9 correct | No previous balance is printed. Day and month are printed as separate pieces of text. |
| [CapEd Credit Union](https://capedcu.com/learn/learning-center/its-a-money-thing/) | Credit card | 9 | Needs review | No | The sample shows page 1 of 2, so the rows cannot reach the summary. |
| [Money Mentors](https://web.archive.org/web/20191114091755/https://moneymentors.ca/schools/programs/high-school/grade-12/credit-paying-for-post-secondary/) (archived page) | Credit card | 7 | All reconcile | 7 of 7 correct | |
| [Niagara University](https://www.niagara.edu/admissions/international-admissions/i-20/) | Savings | 0 | No rows | No | Fails: amounts are printed without cents ("5,000"). |
| [CU*Answers, Black Bars style](https://store.cuanswers.com/product/sage-direct-member-statement-samples/) | Several accounts and a credit card | 55 | Needs review | 55 of 55 correct | All seven account ledgers with activity reconcile. The card is $147.38 short: interest the sample lists without dates. |
| [CU*Answers, Two-Column style](https://store.cuanswers.com/product/sage-direct-member-statement-samples/) | Several accounts and a credit card | 21 | Needs review | No | Fails: two columns of statement side by side on each page are read as one line. |
| [Commerce Bank](https://www.commercebank.com/personal/bank/statement-options) | Checking | 3 | Needs review | 3 of 5 correct | Fails in part: deposit and ATM rows print the date after the description and are missed. The three checks are read. |
| [Boston Architectural College](https://the-bac.edu/info-for/admitted-students/international-admitted-students) | Checking, UK style | 13 | Needs review | 13 of 13 correct | One row flagged where the sample's arithmetic is $3.00 off. |
| [NCUA, MyCreditUnion.gov](https://mycreditunion.gov/brochure-publications/brochure/money-basics-guide-savings-and-checking-accounts) | Checking | 22 | Needs review | 22 of 22 correct | The same sample is printed twice. Its last balance is $100.00 off, and the converter flags it in both copies. |
| [Stanford Federal Credit Union](https://www.sfcu.org/cards/card-center) | Credit card | 6 | All reconcile | 5 of 5 correct | Plus one 0.00 interest line. Descriptions include the reference number column. |

### What was fixed

The failures above led to these changes in the converter, each with a test:

- Statement periods printed with two-digit years ("11/22/22 - 12/21/22", "12/01/19 Thru 12/31/19") now set the year. Before, rows fell back to the current year.
- Rate tables (dated lines with percentages) and check lists (number, date, amount) are no longer read as transactions.
- Layouts with the balance before the description, or the description last, are read.
- Each section's own header names its columns. Statements with several accounts are checked ledger by ledger, from each beginning balance to its ending balance. Loan sections with separate interest and payment columns are supported.
- Credit card payments marked "CR" (glued to the amount or printed apart from it) lower what you owe. Unsigned amounts under titles like "Checks Paid" or "Withdrawals" are money out.
- Day and month printed as separate pieces of text, a sequence number before the date, callout text in a margin beside the table, and a panel printed to the right of the rows are handled.
- Some credit union statements print a hidden "-" after every number. When the column labels carry it too, those signs are ignored and the running balance decides.
- When a reading of the columns would make a statement reconcile only by throwing amounts away, it now loses to the reading that keeps them and flags the rows.

### Data

For each sample, the manifest records the publisher, the PDF's address, the page that links to it, the download date and a SHA-256 checksum. The project also keeps the hand-typed ground truth, the scoring code, the raw per-sample output before and after the fixes, and a script that downloads the set again and verifies the checksums. The PDFs themselves are not redistributed: each stays at its publisher's address.

Download the data:

- [Sample manifest](/tools/statement-pdf-to-csv/benchmark-samples.yaml) (publisher, source URL, linking page, date, SHA-256)
- [Per-sample results after the fixes](/tools/statement-pdf-to-csv/benchmark-results.json) and [before them](/tools/statement-pdf-to-csv/benchmark-results-baseline.json)
- Hand-checked ground truth: [carson-bank-checking](/tools/statement-pdf-to-csv/truth-carson-bank-checking.csv), [cuanswers-black-bars](/tools/statement-pdf-to-csv/truth-cuanswers-black-bars.csv), [money-mentors-credit-card](/tools/statement-pdf-to-csv/truth-money-mentors-credit-card.csv), [ncua-money-basics](/tools/statement-pdf-to-csv/truth-ncua-money-basics.csv), [stanford-fcu-credit-card](/tools/statement-pdf-to-csv/truth-stanford-fcu-credit-card.csv), and, added 7 October, [scott-cu-statement](/tools/statement-pdf-to-csv/truth-scott-cu-statement.csv), [stpaul-grad-ready-bank](/tools/statement-pdf-to-csv/truth-stpaul-grad-ready-bank.csv), [cfpb-credit-card-handout](/tools/statement-pdf-to-csv/truth-cfpb-credit-card-handout.csv) (no rows), [rbfcu-credit-card-lesson](/tools/statement-pdf-to-csv/truth-rbfcu-credit-card-lesson.csv), [commerce-bank-checking](/tools/statement-pdf-to-csv/truth-commerce-bank-checking.csv), [bac-bank-statement-example](/tools/statement-pdf-to-csv/truth-bac-bank-statement-example.csv)
- Converter benchmark data: published with the benchmark results page.

## How the free tools are tested

Every StatementBench tool has automated tests, and they run before every deploy. A tool that fails its tests doesn't ship.

- **CSV to QBO:** fixture CSVs in four bank-export layouts: a newest-first checking export with signed amounts and trailing commas; a day-first export with a preamble, separate debit and credit columns and a currency symbol; an accounting export with month-name dates, parentheses negatives, duplicate rows, a zero-amount row and a total row; and a semicolon-delimited export with decimal commas. The tests check totals to the cent, the date-format choice, stable and unique transaction IDs, and that the generated QBO and OFX files parse back to the same transactions.
- **Statement PDF to CSV:** positioned-text fixtures that mimic what pdf.js returns, in five layouts: a single signed amount column with a running balance over two pages; withdrawal and deposit columns with balances printed once a day; debit and credit columns with no running balance; trailing-minus amounts with month-name dates across New Year; and an unsigned, newest-first layout. They include a deliberately misprinted amount and a misprinted balance, which the tool must flag on exactly the right row. Eleven more fixtures recreate, with made-up names and amounts, the layouts found in the sample statements above: multi-account statements with the description last, card statements with "CR" printed apart, split day and month, margin callouts, hidden minus signs, stacked column labels, check lists after the table, transactions grouped by type, and samples whose own arithmetic is wrong. The accuracy check above is re-run with the same harness.
- **Fee calculator:** plan-selection rules are tested on synthetic price lists, and the live pricing file is validated: every value has a source URL and a date, and unverified values stay "check vendor".

## How pricing data is collected

Prices are read from each vendor's public US pricing page and help center, and recorded with the source URL and date. Promotions are noted but not used in calculations. When a page cannot be fetched, or a value isn't stated, it is recorded as unknown, never filled in from memory or third-party sites. The data is rechecked at least quarterly.

## Corrections

Corrections are dated and described on the page they affect. A correction that changes a benchmark result is also noted at the top of that benchmark page.