# markdown-vs-html-for-llms benchmark

Reproducible char-count and token-count measurements used in the article
`content/blog/{ja,en,es,pt}/markdown-vs-html-for-llms.md`.

## What this measures

A small "What is JSON?" introduction expressed in three forms that contain
the same information:

- `sample.html` — typical CMS-style HTML wrapper (Tailwind-like classes, aria attributes)
- `sample.md` — CommonMark plus the GFM table extension
- `sample.txt` — plain text (markup stripped, table as tab-separated rows)

Each file holds a single h2 heading, a few paragraphs, a three-item bullet
list, a JSON code block, and a three-column table. The three forms convey the
same content; only the surface format differs.

## How to reproduce

```bash
cd scripts/benchmarks/markdown-vs-html-for-llms
python3 -m venv venv
./venv/bin/pip install tiktoken
./venv/bin/python measure.py
```

`measure.py` reports UTF-8 chars, UTF-8 bytes, and token counts under both
`cl100k_base` (GPT-3.5 / GPT-4) and `o200k_base` (GPT-4o family) using the
OpenAI `tiktoken` library.

## Measured results (2026-06-23, tiktoken 0.13.0)

| File           | chars | bytes | cl100k_base tok | o200k_base tok |
| -------------- | ----: | ----: | --------------: | -------------: |
| `sample.html`  | 2,911 | 2,911 |             832 |            835 |
| `sample.md`    | 1,071 | 1,071 |             243 |            247 |
| `sample.txt`   |   986 |   986 |             213 |            217 |

### Reduction vs HTML

- Markdown: chars −63.2 %, cl100k tokens −70.8 %, o200k tokens −70.4 %
- Plain text: chars −66.1 %, cl100k tokens −74.4 %, o200k tokens −74.0 %

## Caveats

- These numbers describe **one specific synthetic document**. Real pages
  differ; external benchmarks (Web2MD, Beam.ai, ReleasePad) report
  reductions from roughly −10 % to −87 % depending on how wrapper-heavy the
  source HTML is.
- Claude and Gemini tokenizers differ from OpenAI's. The direction of the
  difference (HTML wastes tokens on `<`, `>`, `=`, `"`, attribute names) is
  the same for every BPE tokenizer, but the absolute ratios will move.
- Re-running `measure.py` on the same files with the same `tiktoken` version
  must reproduce the numbers above byte-for-byte.

## History

- 2026-06-23 — Created. Replaces fabricated numbers (HTML 1,389 / Markdown
  791 / Plain text 638; tokens 348 / 198 / 160) that appeared in the article
  from its initial commit `8fc6deb` (2026-05-21) onward. The fabricated
  values all had chars / tokens = 3.99 across all three formats, which is
  impossible for a BPE tokenizer because HTML's symbol-heavy content has a
  lower chars-per-token ratio than plain prose.