Benchmarks / yaml-syntax-errors

yaml-syntax-errors

このページは FormatArc のブログ記事が引用している実測の生データです。検索結果には表示されません (noindex)。

Files

README.md

# YAML syntax error messages (4 parsers, measured)

`yaml-syntax-guide` 記事の「よくある間違い × パーサー実エラーメッセージ対応表」の一次ソース。
手書き YAML で頻出する 7 種の書き間違いを主要 4 パーサーに実際に食わせ、**実際に出る
エラーメッセージ文字列・行/列番号の有無・そもそもエラーになるか** を記録する。

`../yaml-parser-differences/` (正常に parse できる曖昧スカラーの型解決差 →
`yaml-json-difference` 記事用) とは測定軸が別: こちらは **不正 YAML の異常系エラー文字列**
のみを扱う。duplicate keys は型解決差ベンチ側でカバー済みのため本ベンチには含めない。

## 測定対象パーサー

| パーサー | バージョン | 実行系 |
| --- | --- | --- |
| js-yaml (`load`, DEFAULT_SCHEMA) | 4.1.1 | Node v26.3.1 |
| yaml (eemeli, `parse` defaults) | 2.8.3 | Node v26.3.1 |
| PyYAML (`safe_load`) | 6.0.3 | Python 3.14.6 |
| ruamel.yaml (`YAML(typ="safe")`) | 0.19.1 | Python 3.14.6 |

実行環境: Apple M5 Pro / macOS (Darwin 25.5.0)。js-yaml は FormatArc の
yaml-to-json ツールが実際に使っているパーサー。

## ケース (cases.json)

1. `tab-indent` — タブでインデント
2. `bad-indent` — 兄弟キーのインデント不揃い (2 vs 3 スペース)
3. `colon-in-value` — 値の中の `: ` をクォートせず二重マッピング化
4. `unclosed-quote` — ダブルクォート閉じ忘れ
5. `undefined-alias` — 未定義アンカーへのエイリアス参照
6. `unclosed-flow` — flow sequence `[...]` の閉じ忘れ
7. `template-brace` — 値の先頭に `{{ }}` テンプレート構文 (Helm / Ansible のレンダリング前ファイル)

## 再現手順

```bash
cd scripts/benchmarks/yaml-syntax-errors

# Node 側 (js-yaml + eemeli yaml) → results-node.json
npm install
node measure.mjs

# Python 側 (PyYAML + ruamel.yaml) → results-python.json
python3 -m venv venv
./venv/bin/pip install pyyaml ruamel.yaml
./venv/bin/python measure.py

# 結合 → results.json
node merge.mjs
```

## 主要な発見 (results.json より)

- 同じ間違いでも 4 パーサーのエラーメッセージは**全て異なる**文言を返す
  (例: タブインデントは js-yaml `tab characters must not be used in indentation` /
  eemeli `Tabs are not allowed as indentation` / PyYAML・ruamel
  `found character '\t' that cannot start any token`)
- `template-brace` は **js-yaml と eemeli yaml ではエラーにならない**。`{{ }}` が flow
  mapping のネストとして解釈され、`{"image":{"[object Object]":null}}` のような壊れた
  オブジェクトが黙って生成される。PyYAML / ruamel は `unhashable key` でエラー
- 行/列番号はほぼ全パーサーが報告するが、eemeli の `undefined-alias`
  (`Unresolved alias`) のみ位置情報なし
- エラー位置の指し方も異なる: `unclosed-quote` で js-yaml は stream 末尾 (L3) を、
  PyYAML はクォート開始位置 (L1:C7) を指す

記事本文の表は `results.json` の値と完全一致させること (数値・文言の捏造禁止ルール準拠)。

results.json

{
  "generatedAt": "see git history",
  "environment": {
    "node": "v26.3.1",
    "python": "3.14.6",
    "machine": "Apple M5 Pro / macOS (Darwin 25.5.0)"
  },
  "cases": [
    {
      "id": "tab-indent",
      "mistake": "Indenting with a tab character instead of spaces",
      "yaml": "parent:\n\tchild: 1",
      "parsers": {
        "js-yaml 4.1.1 load (DEFAULT_SCHEMA)": {
          "outcome": "error",
          "errorName": "YAMLException",
          "message": "tab characters must not be used in indentation (2:1)\n\n 1 | parent:\n 2 | →child: 1\n-----^",
          "line": 2,
          "column": 1
        },
        "yaml (eemeli) 2.8.3 parse (defaults)": {
          "outcome": "error",
          "errorName": "YAMLParseError",
          "message": "Tabs are not allowed as indentation at line 2, column 1:\n\nparent:\n\tchild: 1\n^\n",
          "line": 2,
          "column": 1
        },
        "PyYAML 6.0.3 safe_load": {
          "outcome": "error",
          "errorName": "ScannerError",
          "message": "while scanning for the next token found character '\\t' that cannot start any token in \"<unicode string>\", line 2, column 1: child: 1 ^",
          "line": 2,
          "column": 1
        },
        "ruamel.yaml 0.19.1 YAML(typ='safe')": {
          "outcome": "error",
          "errorName": "ScannerError",
          "message": "while scanning for the next token found character '\\t' that cannot start any token in \"<file>\", line 2, column 1",
          "line": 2,
          "column": 1
        }
      }
    },
    {
      "id": "bad-indent",
      "mistake": "Sibling keys at inconsistent indentation (2 vs 3 spaces)",
      "yaml": "parent:\n  a: 1\n   b: 2",
      "parsers": {
        "js-yaml 4.1.1 load (DEFAULT_SCHEMA)": {
          "outcome": "error",
          "errorName": "YAMLException",
          "message": "bad indentation of a mapping entry (3:5)\n\n 1 | parent:\n 2 |   a: 1\n 3 |    b: 2\n---------^",
          "line": 3,
          "column": 5
        },
        "yaml (eemeli) 2.8.3 parse (defaults)": {
          "outcome": "error",
          "errorName": "YAMLParseError",
          "message": "Nested mappings are not allowed in compact mappings at line 2, column 6:\n\n  a: 1\n     ^\n",
          "line": 2,
          "column": 6
        },
        "PyYAML 6.0.3 safe_load": {
          "outcome": "error",
          "errorName": "ScannerError",
          "message": "mapping values are not allowed here in \"<unicode string>\", line 3, column 5: b: 2 ^",
          "line": 3,
          "column": 5
        },
        "ruamel.yaml 0.19.1 YAML(typ='safe')": {
          "outcome": "error",
          "errorName": "ScannerError",
          "message": "mapping values are not allowed here in \"<file>\", line 3, column 5",
          "line": 3,
          "column": 5
        }
      }
    },
    {
      "id": "colon-in-value",
      "mistake": "Unquoted value containing ': ' (colon + space) — parsed as a second mapping",
      "yaml": "url: http://example.com: 8080",
      "parsers": {
        "js-yaml 4.1.1 load (DEFAULT_SCHEMA)": {
          "outcome": "error",
          "errorName": "YAMLException",
          "message": "bad indentation of a mapping entry (1:24)\n\n 1 | url: http://example.com: 8080\n----------------------------^",
          "line": 1,
          "column": 24
        },
        "yaml (eemeli) 2.8.3 parse (defaults)": {
          "outcome": "error",
          "errorName": "YAMLParseError",
          "message": "Nested mappings are not allowed in compact mappings at line 1, column 6:\n\nurl: http://example.com: 8080\n     ^\n",
          "line": 1,
          "column": 6
        },
        "PyYAML 6.0.3 safe_load": {
          "outcome": "error",
          "errorName": "ScannerError",
          "message": "mapping values are not allowed here in \"<unicode string>\", line 1, column 24: url: http://example.com: 8080 ^",
          "line": 1,
          "column": 24
        },
        "ruamel.yaml 0.19.1 YAML(typ='safe')": {
          "outcome": "error",
          "errorName": "ScannerError",
          "message": "mapping values are not allowed here in \"<file>\", line 1, column 24",
          "line": 1,
          "column": 24
        }
      }
    },
    {
      "id": "unclosed-quote",
      "mistake": "Double-quoted string without a closing quote",
      "yaml": "name: \"unclosed string\nnext: 1",
      "parsers": {
        "js-yaml 4.1.1 load (DEFAULT_SCHEMA)": {
          "outcome": "error",
          "errorName": "YAMLException",
          "message": "unexpected end of the stream within a double quoted scalar (3:1)\n\n 1 | name: \"unclosed string\n 2 | next: 1\n 3 | \n-----^",
          "line": 3,
          "column": 1
        },
        "yaml (eemeli) 2.8.3 parse (defaults)": {
          "outcome": "error",
          "errorName": "YAMLParseError",
          "message": "Missing closing \"quote at line 2, column 8:\n\nnext: 1\n       ^\n",
          "line": 2,
          "column": 8
        },
        "PyYAML 6.0.3 safe_load": {
          "outcome": "error",
          "errorName": "ScannerError",
          "message": "while scanning a quoted scalar in \"<unicode string>\", line 1, column 7: name: \"unclosed string ^ found unexpected end of stream in \"<unicode string>\", line 2, column 8: next: 1 ^",
          "line": 2,
          "column": 8
        },
        "ruamel.yaml 0.19.1 YAML(typ='safe')": {
          "outcome": "error",
          "errorName": "ScannerError",
          "message": "while scanning a quoted scalar in \"<file>\", line 1, column 7 found unexpected end of stream in \"<file>\", line 2, column 8",
          "line": 2,
          "column": 8
        }
      }
    },
    {
      "id": "undefined-alias",
      "mistake": "Alias (*missing) referencing an anchor that was never defined",
      "yaml": "service: *missing",
      "parsers": {
        "js-yaml 4.1.1 load (DEFAULT_SCHEMA)": {
          "outcome": "error",
          "errorName": "YAMLException",
          "message": "unidentified alias \"missing\" (1:18)\n\n 1 | service: *missing\n----------------------^",
          "line": 1,
          "column": 18
        },
        "yaml (eemeli) 2.8.3 parse (defaults)": {
          "outcome": "error",
          "errorName": "ReferenceError",
          "message": "Unresolved alias (the anchor must be set before the alias): missing",
          "line": null,
          "column": null
        },
        "PyYAML 6.0.3 safe_load": {
          "outcome": "error",
          "errorName": "ComposerError",
          "message": "found undefined alias 'missing' in \"<unicode string>\", line 1, column 10: service: *missing ^",
          "line": 1,
          "column": 10
        },
        "ruamel.yaml 0.19.1 YAML(typ='safe')": {
          "outcome": "error",
          "errorName": "ComposerError",
          "message": "found undefined alias 'missing' in \"<file>\", line 1, column 10",
          "line": 1,
          "column": 10
        }
      }
    },
    {
      "id": "unclosed-flow",
      "mistake": "Flow sequence [ ... ] opened but never closed",
      "yaml": "ports: [8080, 9090\nname: web",
      "parsers": {
        "js-yaml 4.1.1 load (DEFAULT_SCHEMA)": {
          "outcome": "error",
          "errorName": "YAMLException",
          "message": "missed comma between flow collection entries (2:1)\n\n 1 | ports: [8080, 9090\n 2 | name: web\n-----^",
          "line": 2,
          "column": 1
        },
        "yaml (eemeli) 2.8.3 parse (defaults)": {
          "outcome": "error",
          "errorName": "YAMLParseError",
          "message": "Flow sequence in block collection must be sufficiently indented and end with a ] at line 2, column 1:\n\nports: [8080, 9090\nname: web\n^\n",
          "line": 2,
          "column": 1
        },
        "PyYAML 6.0.3 safe_load": {
          "outcome": "error",
          "errorName": "ParserError",
          "message": "while parsing a flow sequence in \"<unicode string>\", line 1, column 8: ports: [8080, 9090 ^ expected ',' or ']', but got ':' in \"<unicode string>\", line 2, column 5: name: web ^",
          "line": 2,
          "column": 5
        },
        "ruamel.yaml 0.19.1 YAML(typ='safe')": {
          "outcome": "error",
          "errorName": "ParserError",
          "message": "while parsing a flow sequence in \"<file>\", line 1, column 8 expected ',' or ']', but got ':' in \"<file>\", line 2, column 5",
          "line": 2,
          "column": 5
        }
      }
    },
    {
      "id": "template-brace",
      "mistake": "Template syntax {{ }} at the start of a value (Helm/Ansible before rendering)",
      "yaml": "image: {{ .Values.image }}",
      "parsers": {
        "js-yaml 4.1.1 load (DEFAULT_SCHEMA)": {
          "outcome": "parsed",
          "type": "object",
          "repr": "{\"image\":{\"[object Object]\":null}}"
        },
        "yaml (eemeli) 2.8.3 parse (defaults)": {
          "outcome": "parsed",
          "type": "object",
          "repr": "{\"image\":{\"{ .Values.image }\":null}}"
        },
        "PyYAML 6.0.3 safe_load": {
          "outcome": "error",
          "errorName": "ConstructorError",
          "message": "while constructing a mapping in \"<unicode string>\", line 1, column 8: image: {{ .Values.image }} ^ found unhashable key in \"<unicode string>\", line 1, column 9: image: {{ .Values.image }} ^",
          "line": 1,
          "column": 9
        },
        "ruamel.yaml 0.19.1 YAML(typ='safe')": {
          "outcome": "error",
          "errorName": "ConstructorError",
          "message": "while constructing a mapping in \"<file>\", line 1, column 8 found unhashable key in \"<file>\", line 1, column 9",
          "line": 1,
          "column": 9
        }
      }
    }
  ]
}