Speed up the Markdown parser

[tompng]

The kpeg-generated Markdown parser addresses the input string by character index, so on non-ASCII input every position lookup scans the string from the start and parsing becomes quadratic in the input size — real changelogs (contributor names etc.) always contain some non-ASCII, so this is the default path for large .md files. There were also two smaller costs paid by every document.

Three changes, one per commit:

• Address positions by byte offset. Generated rule bodies only save and restore pos without inspecting it, and all generated regexps are \G-anchored, so overriding the five position helpers (set_string, scan, match_string, get_byte, get_text) with byte-offset versions backed by StringScanner is enough. get_byte (the grammar's .) still consumes one character and returns its codepoint, so the only observable change is the position representation. The override lives in a hand-written file (lib/rdoc/markdown/byte_runtime.rb) prepended from the grammar, so regeneration is unaffected.
• Guard HtmlBlock with a &"<" lookahead. Previously every block position tried ~50 memoized per-tag HTML rules before falling through to Para.
• Skip the Notes pass when the document contains no "[^". parse always ran up to three full passes; a note definition must contain [^, which is rare (11 of 594 sampled markdown files). The References pass is left unconditional: [ appears in almost every real document.

Benchmarks (M-series, Ruby 4.0):

input	before	after
bundle exec rdoc bundler/CHANGELOG.md (5,526 lines)	14.5s	1.5s
300 Japanese paragraphs	2,911ms	40ms
1,000 one-word ASCII paragraphs	160ms	77ms

[matzbot]

🚀 Preview deployment available at: https://93195844.rdoc-6cd.pages.dev (commit: b9fb012)

[Copilot]

Pull request overview

This PR improves RDoc’s kpeg-generated Markdown parser performance on non‑ASCII input by moving the parser runtime from character-indexed positions (which cause repeated O(n) index conversions) to byte-offset positions, and by reducing two always-paid parsing costs in common documents.

Changes:

• Add a ByteRuntime override (backed by StringScanner) to make pos byte-based while preserving character-wise consumption for . (get_byte).
• Optimize HTML block parsing by adding a &"<" lookahead to avoid attempting many HTML-tag rules when the input doesn’t start with <.
• Skip the Notes collection pass when the document does not contain the [^ marker.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
lib/rdoc/markdown/byte_runtime.rb	Introduces a byte-offset runtime override for the kpeg parser helpers (string access, scanning, and substring extraction).
lib/rdoc/markdown.rb	Wires in the byte runtime (via prepend), adds the Notes-pass guard, and includes the HtmlBlock lookahead in the generated parser output.
lib/rdoc/markdown.kpeg	Updates the grammar source to require/prepend the byte runtime, add the Notes-pass guard, and add the HtmlBlock lookahead so regeneration stays consistent.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[skatkov]

Is there a source code for these benchmarks? Curious to see.

[tompng]

Benchmark script

require 'rdoc'

CASES = {
  '300 Japanese paragraphs' =>
    "これはテストです。日本語の**強調**と`コード`を含む段落。\n\n" * 300,
  '1,000 one-word ASCII paragraphs' =>
    "word\n\n" * 1000,
}

CASES.each do |name, code|
  RDoc::Markdown.parse(code) # warmup
  times = 5.times.map do
    GC.start
    t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
    RDoc::Markdown.parse(code)
    Process.clock_gettime(Process::CLOCK_MONOTONIC) - t
  end
  puts format('%-34s %8.1f ms (min of 5)', name, times.min * 1000)
end