Benchmarks / markdown-vs-html-for-llms

markdown-vs-html-for-llms

このページは FormatArc のブログ記事が引用している実測の生データです。検索結果には表示されません (noindex)。

Files

README.md

# markdown-vs-html-for-llms benchmark

Reproducible char-count and token-count measurements used in the article
`content/blog/{ja,en,es,pt}/markdown-vs-html-for-llms.md`.

## What this measures

A small "What is JSON?" introduction expressed in three forms that contain
the same information:

- `sample.html` — typical CMS-style HTML wrapper (Tailwind-like classes, aria attributes)
- `sample.md` — CommonMark plus the GFM table extension
- `sample.txt` — plain text (markup stripped, table as tab-separated rows)

Each file holds a single h2 heading, a few paragraphs, a three-item bullet
list, a JSON code block, and a three-column table. The three forms convey the
same content; only the surface format differs.

## How to reproduce

```bash
cd scripts/benchmarks/markdown-vs-html-for-llms
python3 -m venv venv
./venv/bin/pip install tiktoken
./venv/bin/python measure.py
```

`measure.py` reports UTF-8 chars, UTF-8 bytes, and token counts under both
`cl100k_base` (GPT-3.5 / GPT-4) and `o200k_base` (GPT-4o family) using the
OpenAI `tiktoken` library.

## Measured results (2026-06-23, tiktoken 0.13.0)

| File           | chars | bytes | cl100k_base tok | o200k_base tok |
| -------------- | ----: | ----: | --------------: | -------------: |
| `sample.html`  | 2,911 | 2,911 |             832 |            835 |
| `sample.md`    | 1,071 | 1,071 |             243 |            247 |
| `sample.txt`   |   986 |   986 |             213 |            217 |

### Reduction vs HTML

- Markdown: chars −63.2 %, cl100k tokens −70.8 %, o200k tokens −70.4 %
- Plain text: chars −66.1 %, cl100k tokens −74.4 %, o200k tokens −74.0 %

## Caveats

- These numbers describe **one specific synthetic document**. Real pages
  differ; external benchmarks (Web2MD, Beam.ai, ReleasePad) report
  reductions from roughly −10 % to −87 % depending on how wrapper-heavy the
  source HTML is.
- Claude and Gemini tokenizers differ from OpenAI's. The direction of the
  difference (HTML wastes tokens on `<`, `>`, `=`, `"`, attribute names) is
  the same for every BPE tokenizer, but the absolute ratios will move.
- Re-running `measure.py` on the same files with the same `tiktoken` version
  must reproduce the numbers above byte-for-byte.

## History

- 2026-06-23 — Created. Replaces fabricated numbers (HTML 1,389 / Markdown
  791 / Plain text 638; tokens 348 / 198 / 160) that appeared in the article
  from its initial commit `8fc6deb` (2026-05-21) onward. The fabricated
  values all had chars / tokens = 3.99 across all three formats, which is
  impossible for a BPE tokenizer because HTML's symbol-heavy content has a
  lower chars-per-token ratio than plain prose.

results.json

{
  "measured_at": "2026-06-23",
  "tiktoken_version": "0.13.0",
  "sample_topic": "What is JSON? introduction with h2 + paragraphs + bullet list + code block + 3-column table",
  "samples": {
    "html": {
      "file": "sample.html",
      "chars": 2911,
      "bytes_utf8": 2911,
      "tokens_cl100k_base": 832,
      "tokens_o200k_base": 835,
      "chars_per_cl100k_token": 3.5
    },
    "markdown": {
      "file": "sample.md",
      "format_note": "CommonMark + GFM table extension",
      "chars": 1071,
      "bytes_utf8": 1071,
      "tokens_cl100k_base": 243,
      "tokens_o200k_base": 247,
      "chars_per_cl100k_token": 4.41
    },
    "plain_text": {
      "file": "sample.txt",
      "chars": 986,
      "bytes_utf8": 986,
      "tokens_cl100k_base": 213,
      "tokens_o200k_base": 217,
      "chars_per_cl100k_token": 4.63
    }
  },
  "reduction_vs_html": {
    "markdown": {
      "chars_pct": -63.2,
      "tokens_cl100k_pct": -70.8,
      "tokens_o200k_pct": -70.4
    },
    "plain_text": {
      "chars_pct": -66.1,
      "tokens_cl100k_pct": -74.4,
      "tokens_o200k_pct": -74
    }
  },
  "reproduction_steps": [
    "tmp/markdown-vs-html-benchmark/ 内の sample.{html,md,txt} を見る",
    "python3 -m venv tmp/markdown-vs-html-benchmark/venv",
    "tmp/markdown-vs-html-benchmark/venv/bin/pip install tiktoken",
    "tmp/markdown-vs-html-benchmark/venv/bin/python tmp/markdown-vs-html-benchmark/measure.py"
  ]
}