Benchmarks / csv-to-json-throughput

csv-to-json-throughput

このページは FormatArc のブログ記事が引用している実測の生データです。検索結果には表示されません (noindex)。

Files

README.md

# csv-to-json throughput benchmark

Reproducible CSV-to-JSON throughput numbers used in
`content/blog/{ja,en,es,pt}/csv-to-json-guide.md`.

## What it measures

A 5-column, 1,000,000-row CSV (≈ 52 MB) is converted to JSON / NDJSON
through six approaches and the wall-clock time is recorded for each.

The six approaches:

1. Python stdlib — `csv.DictReader` + `json.dump`
2. pandas — `read_csv` + `to_json(orient="records", lines=True)`
3. Miller — `mlr --c2j cat`
4. Go — `encoding/csv` + `encoding/json` (streaming, `ReuseRecord`)
5. PapaParse streaming — Node, `papaparse` 5.5.2, header mode
6. FormatArc in the browser (not run here; see notes below)

## How to reproduce

```bash
cd scripts/benchmarks/csv-to-json-throughput

# Generate the sample (1M rows × 5 cols)
python3 gen_sample.py 1000000

# Python stdlib + pandas
python3 -m venv venv
./venv/bin/pip install pandas pyarrow
./venv/bin/python bench_python.py
./venv/bin/python bench_pandas.py

# Miller (requires `brew install miller` or equivalent)
mlr --c2j cat sample.csv > out_miller.json

# Go
go run bench_go.go

# PapaParse (Node)
npm install
node bench_papaparse.mjs
```

The FormatArc-in-browser approach is best measured manually: open
`https://formatarc.com/<locale>/csv-to-json/`, paste in `sample.csv`,
and time it. PapaParse is the underlying engine in both Node and
browser builds, but the browser version is bounded by main-thread budget
and the page's responsiveness rather than raw CPU.

## Measured results (2026-06-23, Apple M5 Pro / 64 GB / macOS 26.5)

| Approach | Version | Rows/sec | Wall (s) |
| --- | --- | ---: | ---: |
| Python `csv.DictReader` + `json.dump` | Python 3.14.6 | 232,137 | 4.31 |
| pandas `read_csv` + `to_json` | pandas 3.0.3 | 1,112,836 | 0.90 |
| Miller `mlr --c2j` (warm median of 5) | Miller 6.19.0 | 1,960,784 | 0.51 |
| Go `encoding/csv` + `encoding/json` | Go 1.26.4 | 2,058,201 | 0.49 |
| PapaParse streaming (Node) | papaparse 5.5.2 / Node 26.3.1 | 1,625,566 | 0.62 |

## Caveats

- These numbers are for **one specific machine, one specific row shape,
  one measurement set** (single warm run for Python / pandas / Go /
  PapaParse; warm median of 5 for Miller, where cold-vs-warm difference
  was material). Re-running on a 2024 Intel laptop or a 2021 M1 Air will
  give different (usually lower) numbers.
- pandas 3.x is materially faster than pandas 2.x; Miller 6.x is faster
  than Miller 5.x. Older versions of the same tool can be 2-5× slower.
- CSVs with wider rows, more quoted fields, or non-ASCII text typically
  produce lower throughput than this synthetic ASCII sample.
- Browser performance is hard to compare apples-to-apples with native
  CLIs: the browser shares CPU with the page render thread, V8 startup
  cost dominates small inputs, and large inputs hit memory pressure on
  the page rather than disk swap.

## History

- 2026-06-23 — Created. Replaces a prior "approximate throughput on a
  2024 M3 MacBook" table in the article that had no reproduction script
  and reported numbers 1.5×–6× lower than what the actual tools achieve
  today.

results.json

{
  "measured_at": "2026-06-23",
  "hardware": {
    "chip": "Apple M5 Pro",
    "cores": "18 (6 performance + 12 efficiency)",
    "memory_gb": 64,
    "os": "macOS 26.5 (Darwin 25.5.0, arm64)"
  },
  "sample": {
    "file": "sample.csv",
    "rows": 1000000,
    "columns": 5,
    "size_bytes": 54667810,
    "size_mb": 52.13,
    "schema": [
      "id",
      "name",
      "email",
      "amount",
      "country"
    ]
  },
  "results": {
    "python_stdlib": {
      "tool": "Python csv.DictReader + json.dump",
      "version": "Python 3.14.6 stdlib",
      "rows_per_sec": 232137,
      "wall_seconds": 4.31
    },
    "pandas": {
      "tool": "pandas read_csv + to_json (orient=records, lines=True)",
      "version": "pandas 3.0.3 (Apple Silicon)",
      "rows_per_sec": 1112836,
      "wall_seconds": 0.9
    },
    "miller": {
      "tool": "Miller mlr --c2j cat",
      "version": "Miller 6.19.0 (Go implementation)",
      "rows_per_sec_cold_first_run": 988019,
      "rows_per_sec_warm_median_of_5": 1960784,
      "wall_seconds_cold": 1.01,
      "wall_seconds_warm_median": 0.51,
      "note": "First invocation is roughly half-speed due to FS cache miss on the input file; subsequent warm runs converge to ~1.96M rows/s on this hardware. Throughput reported in the article is the warm median."
    },
    "go": {
      "tool": "Go encoding/csv + encoding/json (streaming, ReuseRecord, 1MB buffer)",
      "version": "Go 1.26.4 darwin/arm64",
      "rows_per_sec": 2058201,
      "wall_seconds": 0.49
    },
    "papaparse_node": {
      "tool": "PapaParse streaming (Node)",
      "version": "papaparse 5.5.2 on Node 26.3.1",
      "rows_per_sec": 1625566,
      "wall_seconds": 0.62
    },
    "browser_formatarc": {
      "tool": "FormatArc CSV to JSON (browser)",
      "version": "PapaParse 5.5.2 in browser",
      "note": "Not measured in this run. The browser implementation loads the full file into memory; the practical ceiling on a recent laptop is in the tens of megabytes / hundreds of thousands of rows before the page becomes unresponsive."
    }
  },
  "caveats": [
    "Numbers are wall-clock for a single warm run. Variance between runs is typically ±5-10%.",
    "Throughput scales with both CPU single-thread speed and tool version. pandas 2.x was substantially slower than 3.x; Miller 5.x was substantially slower than 6.x.",
    "Apple Silicon (M-series) hits much higher throughput than Intel laptops of the same era for csv/JSON workloads.",
    "Real CSVs with wide columns, quoted strings, or unicode content will show different (typically lower) numbers."
  ]
}