Add tests for guarantees L2, L7, L8 - #108
Merged
Merged
Conversation
Nothing parsed markfluence's own written output back through a markdown parser -- what was tested was the encoding mechanism, not the guarantee itself. Feeds every storage2md golden and the larger forward-regression corpus through a fresh StorageToMarkdown conversion (not the static golden text, which would only prove one committed snapshot parses forever) and a real GFM parser, checking that the bracket/table syntax markfluence emitted was actually recognized, plus a bracket-balance sanity check outside fenced code for the case a rendering bug mangles the syntax badly enough that nothing resembling a link is left to flag. Both checks confirmed via mutation testing against a broken destination encoder and a broken link-format string.
No test asserted that page hierarchy is never inferred from directory structure. Added a unit-level case at the most tempting shape to "helpfully" infer from (a file named after the directory its sibling lives one level under) and a batch-level case with files nested several directories deep alongside plausible-looking ancestors, both confirming no parent relationship forms unless a parent: field or --parent said so. Verified by temporarily introducing a directory-based inference into resolveParent and confirming both tests fail.
No test proved either half of the claim -- that root resolution depends only on the files on disk, not on the working directory or on which other files are in the same batch. Added a cwd-independence test (Discover from an unrelated working directory must still resolve an absolute path correctly) and a batch-independence test (resolving one file through a Cache that also resolved an unrelated file from a different project must not change its result). The batch test deliberately uses two distinct projects: an earlier draft shared one project root across both directories, which let a contamination bug that just returns whatever the cache last resolved pass by accident. Both verified by introducing the corresponding bug and confirming the test that should catch it fails.
…just one assertParsesCleanly's bracket-balance check already stripped fenced code before comparing, but the image/link floor count and the table-separator check still ran against the raw source -- so example markdown syntax shown inside a code fence (e.g. "" as documentation text) could inflate the expected count or trip the table check, a false positive this test would blame on the converter instead of on itself. Compute prose once and use it for all three checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes gaps identified in docs/guarantees.md where a guarantee was marked Holds but no test actually proved the specific claim, even where the underlying mechanism was sound (tracked in #107):
output-is-valid-markdown): feeds every storage2md golden and the larger forward-regression corpus through a freshStorageToMarkdownconversion and a real GFM parser, confirming the bracket/table syntax markfluence emitted is actually recognized as such — not the static golden text, which would only prove one committed snapshot parses forever regardless of whether the converter still produces it. Includes a bracket-balance sanity check for the case a rendering bug mangles output so badly nothing resembling a link is left to compare against.no-layout-inference): a unit test and a batch-level test confirming page hierarchy is never inferred from directory nesting, even in the most tempting shape (a file named after a sibling directory).invocation-independent): cwd-independence and batch-independence tests forproject.Discover/Cache— root resolution for a given file must not depend on the process's working directory or on which other files are in the same batch.Every test was verified by deliberately introducing the bug it's meant to catch and confirming it fails, including two cases where that process caught a real flaw in the test's own design before it shipped (a false positive from GFM's "bare space is not a valid destination" rule, and a batch-independence test whose fixture shared one project root across both directories, which would have let a contamination bug pass by accident).
One commit (
38d5e0f) is a self-review fix: the L7 test's fenced-code exclusion was applied inconsistently across its three checks, leaving two of them exposed to false positives from example markdown syntax shown inside a code fence.Test plan
make checkpasses after every commit in this branch