Skip to content

Aligns directive's block_text content to rST's#1048

Closed
kdeldycke wants to merge 1 commit into
executablebooks:masterfrom
kdeldycke:fix-directive-block-text
Closed

Aligns directive's block_text content to rST's#1048
kdeldycke wants to merge 1 commit into
executablebooks:masterfrom
kdeldycke:fix-directive-block-text

Conversation

@kdeldycke

@kdeldycke kdeldycke commented Sep 10, 2025

Copy link
Copy Markdown

In rST, a directive's block_text contains the entire directive, including options and full, unparsed content.

This aligns MyST parser to rST.

Context

I stumbled upon this issue while implementing my own click:example and click:run directives within my Click Extra project: https://kdeldycke.github.io/click-extra/sphinx.html

My implementation is trying to report, within these directives, of the absolute line in the document where a problem occured. The line number and offset calculation works perfectly in rST, but requires some hacks to report the right one in MyST.

But even with these hacks, there is no way to retro-actively compute the absolute line number, because the directive object do not carry its unparsed content.

Example

Compare a given rST directive defined as:

 .. click:run::
    :linenos:

    # This should fail due to variable conflict.
    isolated_filesystem = "Do not overwrite me!"

Its properties are:

>>> print("\n".join(directive.content))
# This should fail due to variable conflict.
isolated_filesystem = "Do not overwrite me!"

>>> print(directive.block_text)
.. click:run::
    :linenos:

    # This should fail due to variable conflict.
    isolated_filesystem = "Do not overwrite me!"

And for its MyST equivalent:

```{click:run}
:linenos:
# This should fail due to variable conflict.
isolated_filesystem = "Do not overwrite me!"
```
>>> print("\n".join(directive.content))
# This should fail due to variable conflict.
isolated_filesystem = "Do not overwrite me!"

>>> print(directive.block_text)
# This should fail due to variable conflict.
isolated_filesystem = "Do not overwrite me!"

Limitations

Note that it is not perfect, as it is supposed to also contain:

  • the complete directive header (i.e the ```{name} {arguments} first line), and
  • the directive closing line (i.e. the ``` line).

But I doubt we can find a way to do it unless we introduce heavier refactor.

Copy link
Copy Markdown
Member

Thanks for this @kdeldycke, and apologies it sat unreviewed for so long.

#1164 now supersedes this: rather than passing the unparsed content only, it reconstructs the full directive text from the fence token — including the opening ```{name} args line and the closing fence that this PR noted it could not include — so block_text should now give you exactly the rST-parity line arithmetic you were after in click-extra, without the workarounds. There's a covering test pinning the reconstructed form.

I'll close this once #1164 is merged.


Generated by Claude Code

chrisjsewell added a commit that referenced this pull request Jul 15, 2026
Fixes three classes of wrong source-line attribution, plus two
rST-parity alignments.

## Directive option warnings point at the offending option

Previously all option warnings pointed at the directive's opening line
(`:opt:` style) or the opening `---` (YAML style), and unknown keys were
aggregated into one warning:

````markdown
```{note}
---
class: tip
name: mynote
align: whatever   <- line 7
---
```
````

reported `<string>:4: 'note': Unknown option keys: ['align'] ...`. Now
each warning (unknown key, invalid value, bad YAML, `#` comments)
carries the offending line — `<string>:7: 'note': Unknown option key:
'align' ...` — for both option styles, resolving the long-standing `TODO
advance line number`:

- `parsers/options.py` gains a public `options_to_tokens` (per-key
`Position`s; `options_to_items` is now a thin wrapper), and its `State`
records comment lines.
- `_parse_directive_options` computes an `options_position` per block
style and per-key absolute lines; the `---` block split is now
**line-based**: text trailing a closing delimiter still becomes body
content (as previously), but is kept as its own line, so it — and every
body line after it — maps exactly (the old character-based split shifted
all subsequent body lines by one).
- Directives synthesized from non-source content (HTML
images/admonitions) attribute option warnings to the element's own line.

## First-line body content was off by one

For no-argument directives with content on the opening line (``
```{note} text ``), every body warning reported one line too low.
`body_offset` is now `-1` for the merged first line (documented in
`DirectiveParsingResult`; all internal consumers verified, and myst-nb's
`code-cell` cannot hit this branch since it takes an argument).

## Stale `document.current_line` (closes #546)

`run_directive` set `document.current_line` and never reset it, so
docutils' `Node.setup_child` stamped the most recent directive's line
(or 0) onto every subsequently-created line-less node — every `Text`
node in the document. This is exactly the "errors attributed to the most
recent directive" / "line 0" behaviour reported for
sphinxcontrib-spelling, and visible in gettext output (table cells
extracted at `index.md:0`). The renderer now keeps `current_line` in
sync as nodes are created; the regenerated `.pot` fixtures show cells at
their real lines.

## rST-parity alignments

- **`block_text`** (supersedes #1048, with thanks to @kdeldycke):
directives now receive the full directive text — opening fence, options
block, unparsed body, closing fence — matching rST, including the
header/closing lines that #1048 noted it could not reconstruct.
Third-party directives that compute absolute lines from `block_text`
(e.g. click-extra) now get rST-consistent input. Note: erroring
directives that embed `self.block_text` in their error output
(`list-table`, `figure`, etc.) now show the full fence rather than only
the body.
- **`include` with `:literal:`**: the literal block's line is now
`start-line + 1` (exact docutils behaviour), not a hardcoded 1.

## Behaviour changes to be aware of

- Unknown-key warnings changed shape: one per key (`Unknown option key:
'x' (allowed: [...])`) instead of one aggregate — the
`myst.directive_option` subtype is unchanged, so suppression configs
keep working.
- Old-vs-new probes were run against master for: the full docutils
rendering suite, a sphinx + sphinx-design project (HTML and
`searchindex.js` byte-identical), myst-nb notebook reading
(byte-identical, including malformed metadata blocks), gettext,
includes, and a ~30k-input fuzz of the option-parsing rewrite — no
rendering differences beyond the intended warning lines/messages and the
two rST-parity items above.

## Known limitations (documented, out of scope)

- Warnings for inline syntax report the enclosing paragraph's first line
(markdown-it inline tokens carry no per-line maps).
- Warnings inside multi-line substitution output can map past the
substitution's location.
- Line attribution inside `{include}`d markdown is off by one
(pre-existing, both before and after this PR) — to be tracked
separately.

All changed fixture numbers were hand-verified against their source
snippets; new conformance cases in `reporter_warnings.md` lock in the
corrected lines (markdown in → warning+line out, reusable as a
language-agnostic corpus).

---
_Generated by [Claude
Code](https://claude.ai/code/session_01GByoptRecR2GAwj5qrLz6C)_

Copy link
Copy Markdown
Member

#1164 is merged, so closing this as superseded — block_text now carries the full directive text including the fence header and closing line. Thanks again for the report and the patch @kdeldycke; the covering test (test_directive_block_text_rst_parity) pins the reconstructed form so it can't silently regress.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants