diff --git a/README.md b/README.md index 70e46a0..be4d690 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Session/tracking data, under `~/interview-prep/` by default: **Per-problem solutions/docs live in four independent, separately configurable directories** — each defaults to a folder under `~/interview-prep/docs/` but is normally pointed at wherever your real practice work lives: - **`DSA_SOLUTIONS_DIR`** (e.g. `~/code/DSA_mock`) — organized as `/.py`, one real Python file per problem grouped by topic, matching a typical personal LeetCode-practice layout. DSA is code, not markdown — see `save_dsa_solution` below. -- **`HLD_SOLUTIONS_DIR`** (e.g. `~/code/HLD`) — one markdown file per HLD problem (e.g. `design-rate-limiter.md`). +- **`HLD_SOLUTIONS_DIR`** (e.g. `~/code/HLD`) — one markdown file per HLD problem (e.g. `design-rate-limiter.md`), plus **`Mock Solutions/`**, the HLD mock loop: one dated folder per attempt holding the reference Claude froze *before* posing the problem, what you actually designed, the diff between them, and the scored evaluation. See [HLD mock interviews](#hld-mock-interviews) below. - **`LLD_SOLUTIONS_DIR`** (e.g. `~/code/LLD`) — holds two separate things: the **reference corpus** of your own designs, one self-contained `.py` per problem grouped into category folders (`1_state_machine/vending_machine.py`), plus flat `design-parking-lot.md` write-ups from `save_practice_doc`; and **`Mock Solutions/`**, the mock-interview loop where you attempt a problem and Claude grades it. See [LLD mock interviews](#lld-mock-interviews) below. A third file, **`DRILL_LOG.md`**, sits at the top level: one running, append-only log of short LLD drills (see [LLD drills](#lld-drills)). - **`BEHAVIORAL_SOLUTIONS_DIR`** (e.g. `~/code/Behavioral`) — holds a single `candidate_context.md`: your reusable background + STAR story bank, with a "Current Focus" section you refresh per company/role. See `save_candidate_context` below. @@ -40,6 +40,13 @@ Session/tracking data, under `~/interview-prep/` by default: | `read_lld_solution` | Open ONE LLD file in full — a past design, a corpus doc, or a mock attempt you're about to grade | | `import_solved_lld_problem(s)` | Backfill the tracker from an LLD design already on disk, *without* touching the file. Refuses anything that isn't `kind=solution` | | `save_lld_solution` | Write a NEW design into the reference corpus, in the right category folder; refuses to overwrite unless told to | +| `start_hld_mock_attempt` | Freeze the grading reference BEFORE posing an HLD problem — creates `Mock Solutions/-/` with `reference.md`. Refuses to overwrite an existing reference; `written_at` comes from the server clock so it can't be backdated | +| `save_hld_attempt` | Capture what you actually designed as `attempt.md` — your near-verbatim turns first, then the interviewer's structured rendering and an explicit list of what you only reached after being prompted | +| `save_hld_diff` | Compare attempt against the frozen reference: matched / **missed invariants** (real gaps) / **diverged at choice points** (fine if the reasoning held), flows as well as boxes. Errors if either side is missing | +| `save_hld_evaluation` | Score an HLD mock against the fixed 7-dimension rubric — writes `evaluation.md`, records the scores and logs the session (so no separate `log_session` for HLD mocks) | +| `list_hld_mock_attempts` | Every HLD attempt folder: which of the four files exist, verdicts, and which attempts were left incomplete | +| `read_hld_mock_file` | Open one of `reference.md` / `attempt.md` / `diff.md` / `evaluation.md` in full | +| `get_hld_feedback` | HLD rubric averages, weakest dimensions, recent verdicts at level — call at session start so the problem choice targets the weak dimensions | | `start_mock_attempt` | Pose an LLD problem — creates `Mock Solutions//` with `problem.md` and an `attempt.py` stub for *you* to fill in. Never overwrites an existing attempt; re-posing opens the next round | | `list_mock_attempts` | Every mock problem and round: which files exist, what's still awaiting grading, scores and verdicts | | `save_mock_evaluation` | Grade an attempt against the fixed 7-dimension rubric — writes `evaluation.md`, logs the session, and regenerates `feedback.md` | @@ -106,6 +113,51 @@ Claude will call `scan_dsa_directory()`, read each file's problem-statement snip `scan_lld_directory()` returns every file with a **Kind** column — `solution` (a per-problem design), `category-doc` (a folder README), `aggregate-doc` (`INDEX.md`, `QUICK_REFERENCE.md` and friends, which span many problems), `practice-doc` (a markdown write-up this server wrote via `save_practice_doc`), `drill-log` (`DRILL_LOG.md`), or `other`. Only `solution` rows are importable; `import_solved_lld_problem(s)` refuses the rest, so an index file can never get linked to a single problem. Claude matches filenames to catalog ids by judgment (`8_lru_cache.py` → `design-lru-cache-oop`), and roughly half the corpus isn't in the 25-entry built-in LLD catalog at all (`order_lifecycle.py`, `whatsapp_messaging.py`, `audit_trail.py`, …) — those need `add_custom_problem` first. Nothing under `~/code/LLD` is modified; only `index.json` is written. +## HLD mock interviews + +Same loop as LLD, with one structural difference: **the reference is written and frozen before you're asked the question.** Claude generates its grading reference, `start_hld_mock_attempt` puts it on disk, and only then does the interview start. A reference that still lives in the model's context can drift under your pushback; one on disk can't. + +``` +~/code/HLD/Mock Solutions/ + 2026-08-21-design-distributed-task-scheduler/ + reference.md # frozen BEFORE the problem was posed; written_at is the server's clock + attempt.md # what you actually designed, warts intact + diff.md # matched / missed invariants / diverged at choice points + evaluation.md # verdict, verdict-at-level, seven-dimension scorecard +``` + +One folder per *attempt*, named `YYYY-MM-DD-`; a second attempt at the same problem on the same day is `-r2`. Attempting a problem again never lands on the earlier evidence. + +**The flow.** Say: + +> Let's do a mock HLD interview. + +1. Claude calls `get_hld_feedback()` and `get_catalog("HLD")`, picks a problem aimed at your weakest dimensions, generates its reference, and calls `start_hld_mock_attempt(...)` — **before** giving you the prompt. +2. You do the interview. +3. At the end: `save_hld_attempt(...)` → `save_hld_diff(...)` → `save_hld_evaluation(...)` → `save_practice_doc(...)`. + +**Attempt vs. practice doc.** The attempt folder holds the evidence, ungroomed; the practice doc holds the correct design. Keeping them apart is the whole point — an attempt rewritten toward the right answer reads clean six weeks later and tells you nothing about which parts you actually got right. Nothing under `Mock Solutions/` is ever surfaced by `list_practice_docs`. + +**`raw_turns` is the load-bearing argument** of `save_hld_attempt`. Written from memory instead, the attempt gets unconsciously tidied — a justification you never gave gets filled in, an explanation that doubled back gets straightened — and the diff then grades a cleaned-up version of what happened. `attempt.md` also carries an explicit *arrived-at-only-after-prompting* list: what you reached only once the gap was named, with the prompting question quoted. + +**Missed vs. diverged.** `save_hld_diff` separates a missing reference **invariant** (a real gap) from a different option taken at a **choice point** (not a gap if the trade-off reasoning was sound). The diff compares end-to-end flows too, not just which components got named — two designs can list identical boxes and route a request completely differently. + +**The rubric**, seven dimensions, 1-5, matching the `hld-interviewer` skill's scorecard, stored under `hld:`-prefixed keys in `index.json`: + +| Dimension | What it measures | +|---|---| +| `requirements` | Functional/non-functional, what got pinned down | +| `capacity-estimation` | Whether you justified the numbers yourself | +| `architecture` | Component design, and whether each box is warranted | +| `deep-dives` | Depth on the hard parts under drilling | +| `scale-calibration` | Reading the stated numbers vs. pattern-matching to FAANG scale | +| `communication` | Structure, signposting, driving the session | +| `composure` | Behaviour under pushback and persona pressure | + +Free-form keys are rejected — the aggregate is the whole value, and keys that vary per session never aggregate. + +**End-to-end flows.** Every HLD design document — `reference.md` and anything `save_practice_doc` writes — should carry an `## End-to-end flows` section right after the architecture diagram: at least three numbered flows (write path, primary read-or-execute path, failure/recovery path), each step naming the component, the operation and the datastore touched. A diagram shows what exists; this shows what happens. Its absence is a **warning in the tool result, not an error** — the doc still gets written. + ## LLD mock interviews The loop that turns practice into targeted practice. You write the design yourself; Claude grades it as the interviewer and remembers where you're weak. @@ -171,7 +223,7 @@ get_lld_drill_log(limit=5) # limit=0 returns the whole file Say this at the start of a chat (or bake it into a Claude Desktop project/skill): -> At the start of every practice session, call `get_progress_summary`, then `suggest_next_problems` for the type we're practicing (DSA/HLD/LLD) and let me pick from the top few. For LLD also call `get_lld_feedback` first and aim the pick at my weakest rubric dimensions. After we solve/discuss it, call `log_session` (with `problem_id` set). Then for HLD/LLD call `save_practice_doc` with the full write-up (HLD: requirements, capacity estimate, architecture, API/data model, trade-offs; LLD: class design, patterns used, key decisions). For DSA, if the problem isn't already on disk, call `save_dsa_solution` with the final code and a short explanation. +> At the start of every practice session, call `get_progress_summary`, then `suggest_next_problems` for the type we're practicing (DSA/HLD/LLD) and let me pick from the top few. For LLD also call `get_lld_feedback` first and aim the pick at my weakest rubric dimensions; for HLD call `get_hld_feedback` first for the same reason. After we solve/discuss it, call `log_session` (with `problem_id` set). Then for HLD/LLD call `save_practice_doc` with the full write-up (HLD: requirements, capacity estimate, architecture, API/data model, trade-offs; LLD: class design, patterns used, key decisions). For DSA, if the problem isn't already on disk, call `save_dsa_solution` with the final code and a short explanation. > > If I say I want to *attempt* an LLD problem myself rather than discuss it, run the mock loop instead: `start_mock_attempt`, wait for me to write `attempt.py`, then `read_lld_solution`, `save_mock_evaluation` (score honestly — inflated scores break the feedback loop), `save_ideal_solution`, and `save_simple_solution` for the cut-down version. @@ -211,9 +263,10 @@ A stdio-transport MCP server (what this is) isn't a background daemon you start Four ways to check it's actually working, in increasing order of realism: 1. **Does it even boot?** `python3 server.py` from this folder — should start and hang silently (that's correct; it's waiting for a client on stdin). Ctrl+C to stop. -2. **Does it speak MCP correctly?** `python3 test_server.py` — spawns the server as a real MCP client would, does the protocol handshake, lists all 29 tools, and calls `get_progress_summary` for real. Read-only, safe to run anytime. A clean "All checks passed" means the server itself is solid, independent of Claude Desktop. +2. **Does it speak MCP correctly?** `python3 test_server.py` — spawns the server as a real MCP client would, does the protocol handshake, lists all 37 tools, and calls `get_progress_summary` for real. Read-only, safe to run anytime. A clean "All checks passed" means the server itself is solid, independent of Claude Desktop. 3. **Do the LLD tools actually behave?** `python3 test_lld_tools.py` — builds a synthetic design repo in a temp directory and exercises every LLD tool against it: path guards, kind classification, rubric validation, the full mock loop, and above all that `attempt.py` comes out byte-identical to what was written. Hermetic — it never touches `~/code/LLD` or your real `index.json`. -4. **Is Claude Desktop actually using it?** +4. **Do the HLD mock tools actually behave?** `python3 test_hld_tools.py` — builds a synthetic HLD root in a temp directory and walks the acceptance checklist: a frozen `reference.md` can't be overwritten, `written_at` comes from the server clock, a diff with only one side errors, a rubric key outside the fixed seven is rejected, `Mock Solutions/` never leaks into `list_practice_docs`, a doc with no `## End-to-end flows` warns but still writes, and `get_hld_feedback` on an empty history returns cleanly. Hermetic — it never touches `~/code/HLD` or your real `index.json`. +5. **Is Claude Desktop actually using it?** - Open a chat and look at the tools/connectors icon near the input box — `interview-memory` should be listed with its tool count. - `ps aux | grep server.py` — while Desktop is open, you should see a live `python3 .../server.py` process (Desktop spawns it once you open a chat that uses it, or at startup depending on version). - Logs: `~/Library/Logs/Claude/mcp-server-interview-memory.log`. @@ -226,8 +279,9 @@ This server only ever runs over **stdio** (`mcp.run(transport="stdio")`), the sa Within the local filesystem, a few guardrails are worth knowing about since Claude drives these tools somewhat autonomously: - **Path sanitization.** Every value used to build a filename (`problem_id`, problem titles) is passed through a slugifier that strips everything except `a-z 0-9 -` before it ever touches a path. A value like `../../etc/passwd` becomes `etc-passwd`, not a traversal — verified with an adversarial test during development. -- **Directory containment.** `save_practice_doc` can only write inside `HLD_SOLUTIONS_DIR` or `LLD_SOLUTIONS_DIR` (whichever matches the call); `save_dsa_solution` can only write inside `DSA_SOLUTIONS_DIR`; `save_candidate_context`/`get_candidate_context` are confined to `BEHAVIORAL_SOLUTIONS_DIR`; `import_solved_dsa_problem(s)` can only *link* to files that already live inside `DSA_SOLUTIONS_DIR` (it refuses anything outside it, e.g. `~/.ssh/`, `/etc/`); `get_practice_doc` re-checks that whatever `index.json` points at is still inside the allowed directory for that problem type before reading it. +- **Directory containment.** `save_practice_doc` can only write inside `HLD_SOLUTIONS_DIR` or `LLD_SOLUTIONS_DIR` (whichever matches the call); the HLD mock tools take a caller-supplied `attempt_folder` and resolve it against `HLD_SOLUTIONS_DIR/Mock Solutions/`, refusing anything that lands outside it (and `read_hld_mock_file` only accepts the four known filenames, so `filename` can't name anything else); `save_dsa_solution` can only write inside `DSA_SOLUTIONS_DIR`; `save_candidate_context`/`get_candidate_context` are confined to `BEHAVIORAL_SOLUTIONS_DIR`; `import_solved_dsa_problem(s)` can only *link* to files that already live inside `DSA_SOLUTIONS_DIR` (it refuses anything outside it, e.g. `~/.ssh/`, `/etc/`); `get_practice_doc` re-checks that whatever `index.json` points at is still inside the allowed directory for that problem type before reading it. - **No code execution.** The server only reads and writes text files. It never `exec`s, `eval`s, or runs the DSA solutions it stores — `code`/`content_markdown` arguments are treated purely as bytes to persist. +- **Pre-commitment protection.** `start_hld_mock_attempt` refuses to overwrite an existing `reference.md` outright — there is no `overwrite` flag, because a reference that can be rewritten after the candidate has spoken isn't a reference. A genuine re-attempt opens a new `-r2` folder instead. Its `written_at` comes from the server's own clock, not a tool argument, so it can't be backdated. - **Overwrite protection.** `save_dsa_solution` refuses to replace an existing file unless `overwrite=True` is passed explicitly, since `DSA_SOLUTIONS_DIR` is assumed to hold real, hand-written work. (`save_practice_doc` for HLD/LLD, and `save_candidate_context` for Behavioral, do overwrite by design — they hold Claude's latest write-up/profile, and those directories are managed entirely by this server.) - **Env-var-scoped roots.** `INTERVIEW_PREP_DIR`, `DSA_SOLUTIONS_DIR`, `HLD_SOLUTIONS_DIR`, `LLD_SOLUTIONS_DIR`, and `BEHAVIORAL_SOLUTIONS_DIR` are only ever set by you, in your own Claude Desktop config — they aren't something a chat message can override. diff --git a/server.py b/server.py index 45d8a1d..00b6c82 100644 --- a/server.py +++ b/server.py @@ -21,6 +21,13 @@ each point at an existing personal repo instead of living inside INTERVIEW_PREP_DIR: HLD_SOLUTIONS_DIR/.md One markdown file per HLD problem. + HLD_SOLUTIONS_DIR/Mock Solutions/-/ + The HLD mock loop: reference.md (written and + frozen BEFORE the problem is posed), + attempt.md (what the candidate actually + produced), diff.md and evaluation.md. One + folder per attempt, so a re-attempt never + lands on the earlier evidence. LLD_SOLUTIONS_DIR//.py The LLD reference corpus: one self-contained .py design per problem, grouped by category @@ -76,6 +83,22 @@ The rubric vocabulary is fixed (LLD_RUBRIC) so scores aggregate across sessions -- that aggregate is what step 1 reads, closing the loop. +HLD mock-interview loop (Claude interviews, then grades against a reference +it committed to before the candidate spoke): + 1. get_hld_feedback() -- which of the seven dimensions are weakest + 2. get_catalog("HLD") -- pick a problem that forces them + 3. start_hld_mock_attempt(...) -- freeze the reference BEFORE posing the + problem; it can't be rewritten after + 4. ... run the interview ... + 5. save_hld_attempt(...) -- the candidate's design, raw turns first + 6. save_hld_diff(...) -- matched / missed invariants / diverged at + choice points, flows as well as boxes + 7. save_hld_evaluation(...) -- score it; this also logs the session + 8. save_practice_doc(...) -- the clean revision artifact +The attempt folder holds the evidence, ungroomed; the practice doc holds the +correct design. Keeping them apart is the point -- an attempt rewritten to the +right answer is useless for revision six weeks later. + LLD drills (short focused reps, no rubric, no per-problem folder): 1. get_lld_drill_log() -- what was drilled recently, what's unresolved 2. ... run the drill with the user ... (get_current_time at both ends @@ -186,6 +209,43 @@ ] LLD_RUBRIC_PREFIX = "lld:" +# --- HLD: mock-attempt folders (the pre-committed reference + the diff) ------ +# HLD mirrors the LLD mock loop, but the artifact is markdown rather than code +# and the folder is per-ATTEMPT rather than per-problem: one dated folder holds +# reference.md (written and frozen BEFORE the problem is posed), attempt.md +# (what the candidate actually produced), diff.md and evaluation.md. Excluded +# from list_practice_docs the same way the LLD one is -- an attempt is evidence +# of what happened under time pressure, not revision material. +HLD_MOCK_DIRNAME = "Mock Solutions" +HLD_MOCK_DIR = HLD_SOLUTIONS_DIR / HLD_MOCK_DIRNAME +# The only filenames these tools will read or write inside an attempt folder. +# read_hld_mock_file uses this as its whitelist, so a caller-supplied filename +# can never name anything else. +HLD_MOCK_FILES = ("reference.md", "attempt.md", "diff.md", "evaluation.md") + +# The seven-dimension HLD scorecard, matching the hld-interviewer skill. Fixed +# for the same reason LLD_RUBRIC is: free-form keys never aggregate, and the +# aggregate (get_hld_feedback) is what makes the next session target the weak +# dimensions instead of picking blind. Stored under an "hld:" prefix so they +# share index["competency_scores"] with the LLD rubric without colliding -- +# "requirements" here and "requirements-and-scope" there are different scales. +HLD_RUBRIC = [ + "requirements", + "capacity-estimation", + "architecture", + "deep-dives", + "scale-calibration", + "communication", + "composure", +] +HLD_RUBRIC_PREFIX = "hld:" + +# Every HLD design document must narrate how a request actually moves through +# the system, not just which boxes exist. A component diagram shows what +# exists; this section shows what happens. Its absence is warned about (never +# blocked) when a doc or a reference is written. +FLOWS_HEADING = "## End-to-end flows" + VALID_TYPES = {"HLD", "LLD", "DSA", "Behavioral", "Other"} CATALOG_TYPES = {"DSA", "HLD", "LLD", "Behavioral"} VALID_VERDICTS = {"Strong Hire", "Hire", "Lean Hire", "Lean No Hire", "No Hire"} @@ -602,32 +662,108 @@ def _ensure_drill_log() -> None: ) -def _rubric_scores() -> dict: - """The LLD rubric slice of index["competency_scores"], keys un-prefixed.""" +def _rubric_scores(prefix: str = LLD_RUBRIC_PREFIX) -> dict: + """One rubric's slice of index["competency_scores"], keys un-prefixed. The + LLD and HLD rubrics share that dict under different prefixes, so every + reader here is scoped by prefix; the defaults keep the LLD callers reading + the LLD rubric.""" scores = _load_index().get("competency_scores", {}) - return { - k[len(LLD_RUBRIC_PREFIX):]: v - for k, v in scores.items() - if k.startswith(LLD_RUBRIC_PREFIX) - } + return {k[len(prefix):]: v for k, v in scores.items() if k.startswith(prefix)} -def _rated_dimensions() -> List[tuple]: +def _rated_dimensions(prefix: str = LLD_RUBRIC_PREFIX) -> List[tuple]: """(dimension, avg, count) for every scored rubric dimension, worst first.""" - rated = [(k, v["avg"], v["count"]) for k, v in _rubric_scores().items() if v.get("count")] + rated = [(k, v["avg"], v["count"]) for k, v in _rubric_scores(prefix).items() if v.get("count")] rated.sort(key=lambda kv: kv[1]) return rated -def _unrated_dimensions() -> List[str]: +def _unrated_dimensions(rubric: List[str] = LLD_RUBRIC, prefix: str = LLD_RUBRIC_PREFIX) -> List[str]: """Rubric dimensions no attempt has exercised yet.""" - scored = _rubric_scores() - return [d for d in LLD_RUBRIC if d not in scored] + scored = _rubric_scores(prefix) + return [d for d in rubric if d not in scored] -def _weakest_dimensions(limit: int = 3) -> List[tuple]: +def _weakest_dimensions(limit: int = 3, prefix: str = LLD_RUBRIC_PREFIX) -> List[tuple]: """The lowest-scoring rubric dimensions — what the next mock should target.""" - return _rated_dimensions()[:limit] + return _rated_dimensions(prefix)[:limit] + + +# --------------------------------------------------------------------------- +# Helpers: HLD mock-attempt folders +# --------------------------------------------------------------------------- + +def _hld_attempt_folder(problem_id: str, round_no: int = 1) -> Path: + """Mock Solutions/-[-r]/ -- one folder per ATTEMPT, so + re-attempting the same problem later never lands on the earlier evidence. + The slug always goes through _slugify, so a problem_id like "../../etc" + can't escape the mock directory.""" + suffix = "" if round_no <= 1 else f"-r{round_no}" + return HLD_MOCK_DIR / f"{date.today().isoformat()}-{_slugify(problem_id)}{suffix}" + + +def _resolve_hld_folder(attempt_folder: str) -> "Path | None": + """Resolve a caller-supplied attempt folder -- either a bare folder name as + returned by start_hld_mock_attempt, or its absolute path -- to a path + inside HLD_MOCK_DIR. None if it points anywhere else, which is what stops + "../../.ssh" or an absolute path elsewhere on disk being read or written + through these tools. Returns the path whether or not it exists; callers + check that themselves so they can say which file is missing.""" + raw = Path(attempt_folder.strip()).expanduser() + candidate = raw if raw.is_absolute() else HLD_MOCK_DIR / raw + try: + resolved = candidate.resolve() + root = HLD_MOCK_DIR.resolve() + except OSError: + return None + # The mock root itself is not an attempt folder. + if resolved == root or not resolved.is_relative_to(root): + return None + return resolved + + +def _hld_folder_meta(folder: Path) -> dict: + """Everything derivable about one attempt folder: the date, problem id and + round encoded in its name, plus title/variant_of read back out of + reference.md's frontmatter. Used by the evaluation and listing tools so the + folder name stays the single source of truth for what an attempt is about.""" + name = folder.name + stamp, rest = name[:10], name[11:] # "YYYY-MM-DD" + "-" + the rest + round_no = 1 + match = re.search(r"-r(\d+)$", rest) + if match: + round_no = int(match.group(1)) + rest = rest[: match.start()] + meta = {"folder": name, "date": stamp, "problem_id": rest, "round": round_no, + "title": "", "variant_of": ""} + + reference = folder / "reference.md" + if reference.exists(): + try: + head = reference.read_text(encoding="utf-8", errors="ignore")[:1200] + except OSError: + head = "" + for line in head.splitlines(): + for key in ("title", "variant_of"): + if line.startswith(f"{key}: "): + meta[key] = line[len(key) + 2:].strip() + return meta + + +def _flows_warning(markdown: str, what: str) -> str: + """A warning (never an error) when a design document doesn't narrate its + end-to-end flows. The write always succeeds -- the point is to catch the + doc that shows what exists but never what happens, at the moment it's + written rather than six weeks later during revision.""" + if FLOWS_HEADING.lower() in markdown.lower(): + return "" + return ( + f"\n\nWARNING: {what} has no `{FLOWS_HEADING}` section. Add one after " + "the architecture diagram and before the deep dives, with at least " + "three numbered flows (the write path, the primary read-or-execute " + "path, and a failure/recovery path). Each step should name the " + "component, the operation and the datastore it touches." + ) def _mock_history_rows(records: List[dict], limit: int = 0) -> List[tuple]: @@ -730,21 +866,29 @@ def get_progress_summary() -> str: due = sum(1 for info in tracker.values() if _days_since(info["last_practiced"]) >= STALE_DAYS) lines.append(f"- {t}: {attempted}/{total} catalog problems attempted, {due} due for revision") - # Behavioral competencies and LLD rubric dimensions share this dict but are + # Behavioral competencies and the two mock rubrics share this dict but are # different scales measuring different things, so they're reported apart -- - # LLD keys carry the LLD_RUBRIC_PREFIX (see save_mock_evaluation). + # the rubric keys carry LLD_RUBRIC_PREFIX / HLD_RUBRIC_PREFIX (see + # save_mock_evaluation / save_hld_evaluation). Anything unprefixed is + # behavioral, so a new rubric must be excluded here as well as reported + # below, or its scores show up as behavioral competencies. competency_scores = index.get("competency_scores", {}) - behavioral = {k: v for k, v in competency_scores.items() if not k.startswith(LLD_RUBRIC_PREFIX)} + rubric_prefixes = (LLD_RUBRIC_PREFIX, HLD_RUBRIC_PREFIX) + behavioral = {k: v for k, v in competency_scores.items() if not k.startswith(rubric_prefixes)} if behavioral: lines += ["", "Competency scores (behavioral, 1-5 avg):"] for area, stats in sorted(behavioral.items(), key=lambda kv: kv[1]["avg"]): lines.append(f"- {area}: {stats['avg']:.1f} ({stats['count']} rated)") - rated = _rated_dimensions() - if rated: - lines += ["", "LLD mock rubric (1-5 avg, weakest first):"] - lines += [f"- {dim}: {avg:.1f} ({n} rated)" for dim, avg, n in rated] - lines.append("Call get_lld_feedback before an LLD session for the full scorecard.") + for label, prefix, tool in ( + ("LLD", LLD_RUBRIC_PREFIX, "get_lld_feedback"), + ("HLD", HLD_RUBRIC_PREFIX, "get_hld_feedback"), + ): + rated = _rated_dimensions(prefix) + if rated: + lines += ["", f"{label} mock rubric (1-5 avg, weakest first):"] + lines += [f"- {dim}: {avg:.1f} ({n} rated)" for dim, avg, n in rated] + lines.append(f"Call {tool} before an {label} session for the full scorecard.") return "\n".join(lines) @@ -1136,9 +1280,16 @@ def save_practice_doc(problem_type: str, title: str, content_markdown: str, prob Write the ENTIRE content yourself in content_markdown — this tool only persists it, it doesn't generate anything. Suggested shape per type: - HLD: requirements (functional/non-functional), capacity estimation, - high-level architecture (components as text/ASCII diagram), API - design, data model, deep dives on the hard parts, trade-offs, what - you'd change under different constraints. + high-level architecture (components as text/ASCII diagram), an + `## End-to-end flows` section, API design, data model, deep dives on + the hard parts, trade-offs, what you'd change under different + constraints. The flows section is required and comes immediately + after the architecture diagram: at least three numbered, sequential + flows -- the write path, the primary read-or-execute path, and a + failure/recovery path -- each step naming the component, the + operation and the datastore touched ("sweeper runs `SELECT ... FOR + UPDATE SKIP LOCKED` on `job_runs`", not "sweeper picks up jobs"). A + diagram shows what exists; this shows what happens. - LLD: problem framing, key entities/classes with responsibilities, class diagram (as text), design patterns used and why, key design decisions and trade-offs, extensibility notes. @@ -1197,7 +1348,11 @@ class diagram (as text), design patterns used and why, key design } _save_index(index) - return f"Saved {problem_type} doc for `{slug}` to {path}." + # HLD docs are the revision artifact, so they must narrate what happens, + # not just what exists. Warned about rather than blocked: the doc is worth + # keeping either way, and refusing the write would lose it. + warning = _flows_warning(content_markdown, "this HLD doc") if problem_type == "HLD" else "" + return f"Saved {problem_type} doc for `{slug}` to {path}." + warning @mcp.tool() @@ -2297,5 +2452,562 @@ def get_lld_drill_log(limit: int = 5) -> str: return heading + ".\n\n" + LLD_DRILL_SEPARATOR.join(shown).strip() +# --------------------------------------------------------------------------- +# Tools: the HLD mock loop (pre-committed reference -> attempt -> diff -> score) +# --------------------------------------------------------------------------- + +@mcp.tool() +def start_hld_mock_attempt( + problem_id: str, + title: str, + reference_markdown: str, + variant_of: str = "", + difficulty: str = "", + round_no: int = 1, +) -> str: + """Freeze your grading reference BEFORE posing an HLD problem. Creates + "Mock Solutions/-/" under the HLD root and writes + reference.md into it, stamped with the server's own clock. + + Call this at the START of a mock HLD interview, after generating the + reference and BEFORE giving the candidate the prompt. That ordering is the + entire point: a reference that still lives only in your context can drift + under candidate pushback, and a reference you can revise mid-session isn't + a reference. Once this returns, it is on disk and this tool will not + rewrite it. + + Structure reference_markdown as the hld-interviewer skill specifies — + invariants, choice points, scale verdict, the delta table for variants — + and include a `## End-to-end flows` section, since save_hld_diff compares + flows and not just which components got named. + + Refuses to overwrite an existing reference.md. To attempt the same problem + again on the same day, pass round_no=2 (folder suffix "-r2"), which is a + new attempt rather than a rewritten reference. + + Args: + problem_id: Catalog id (see get_catalog) or a slug for a custom + problem. For a variant, use the variant's own id, not the + canonical's. + title: Problem title as posed, e.g. "Design a Distributed Task Scheduler". + reference_markdown: The full frozen reference, authored entirely by + you before the candidate has said anything about the design. + variant_of: Canonical problem_id this is a variant of, if any. + difficulty: Optional Easy/Medium/Hard. + round_no: 2, 3, ... to open another attempt at the same problem on the + same day. Default 1. + + Returns the attempt folder path — pass it to save_hld_attempt, + save_hld_diff and save_hld_evaluation for the rest of the session. + """ + slug = _slugify(problem_id) if problem_id.strip() else _slugify(title) + if not slug: + return "problem_id/title produced an empty id — give the problem a name." + if not reference_markdown.strip(): + return "reference_markdown is empty — the reference is the pre-commitment." + + folder = _hld_attempt_folder(slug, round_no) + if not _resolve_hld_folder(str(folder)): + return f"Refusing to write outside {HLD_MOCK_DIR}." + + reference = folder / "reference.md" + if reference.exists(): + return ( + f"{reference} already exists — refusing to overwrite a frozen " + "reference. The pre-commitment only binds if it can't be rewritten " + "after the candidate has spoken. If this is genuinely a new " + f"attempt at `{slug}`, call again with round_no=2 (or the next " + f"free number) to open a separate folder alongside {folder.name}." + ) + + folder.mkdir(parents=True, exist_ok=True) + front = ["---", f"problem_id: {slug}", f"title: {title.strip()}"] + if variant_of.strip(): + front.append(f"variant_of: {_slugify(variant_of)}") + if difficulty.strip(): + front.append(f"difficulty: {difficulty.strip()}") + # The server's own clock, never a tool argument: a reference whose + # written_at could be supplied by the caller could be backdated, and a + # backdated pre-commitment is not a pre-commitment. + front += [f"written_at: {_now().strftime('%Y-%m-%dT%H:%M:%S')}", "frozen: true", "---", ""] + reference.write_text( + "\n".join(front) + f"\n# {title.strip()} — reference (frozen)\n\n" + + reference_markdown.strip() + "\n", + encoding="utf-8", + ) + + return ( + f"Reference frozen for `{slug}` (round {round_no}).\n" + f"- Attempt folder: {folder}\n" + f"- Reference: {reference}\n\n" + "Now pose the problem. At the end of the session: save_hld_attempt → " + "save_hld_diff → save_hld_evaluation → save_practice_doc." + + _flows_warning(reference_markdown, "reference.md") + ) + + +@mcp.tool() +def save_hld_attempt(attempt_folder: str, attempt_markdown: str, raw_turns: str = "") -> str: + """Capture what the candidate ACTUALLY designed, as attempt.md inside the + attempt folder start_hld_mock_attempt created. Call this at the end of the + interview, before save_hld_diff. + + raw_turns matters more than attempt_markdown. Pass the candidate's design + turns close to verbatim. Authoring this from memory instead unconsciously + tidies the reasoning — filling in a justification they never gave, + straightening out an explanation that doubled back — and that silently + destroys the diff, which is the only reason the attempt is saved + separately at all. + + This tool writes the "As stated by the candidate" section from raw_turns + and then your attempt_markdown, which must supply the other two sections: + + ## Interviewer's structured rendering + The same design normalised into the standard section shape, so it can be + compared mechanically against the reference. + + ## Arrived-at-only-after-prompting + Every conclusion the candidate reached AFTER you named the gap, with the + prompting question quoted. Keep this a first-class list rather than a + remark buried in prose — the distinction between "said it" and "said it + once asked" is load-bearing in this candidate's feedback. + + The attempt is a record of what was produced under time pressure. Do not + correct it toward the reference; correct material belongs in reference.md + and in the practice doc. + + Args: + attempt_folder: The folder returned by start_hld_mock_attempt (name or + full path). + attempt_markdown: The structured rendering plus the + arrived-at-only-after-prompting list, authored by you. + raw_turns: The candidate's own words, lightly cleaned for + transcription noise only. + """ + folder = _resolve_hld_folder(attempt_folder) + if not folder: + return f"attempt_folder must name a folder inside {HLD_MOCK_DIR}." + if not folder.is_dir(): + return ( + f"No attempt folder at {folder}. Use list_hld_mock_attempts to see " + "what exists, or start_hld_mock_attempt to open one." + ) + if not attempt_markdown.strip(): + return "attempt_markdown is empty — nothing to save." + + meta = _hld_folder_meta(folder) + display_title = meta["title"] or meta["problem_id"] + path = folder / "attempt.md" + today = date.today().isoformat() + + stated = raw_turns.strip() or ( + "_(not captured — the interviewer did not pass raw_turns, so this " + "attempt has no verbatim record and the diff below rests on a " + "reconstruction.)_" + ) + path.write_text( + f"# {display_title} — candidate attempt\n\n" + f"_Saved: {today} · id: `{meta['problem_id']}` · folder: `{folder.name}`_\n\n" + "---\n\n" + "## As stated by the candidate\n\n" + "_Transcribed from the session, lightly cleaned. No reasoning added._\n\n" + f"{stated}\n\n" + "---\n\n" + + attempt_markdown.strip() + "\n", + encoding="utf-8", + ) + + warnings = [] + if not raw_turns.strip(): + warnings.append( + "raw_turns was empty, so there is no verbatim record of what the " + "candidate said — the diff is now grading a reconstruction." + ) + for heading in ("## Interviewer's structured rendering", "## Arrived-at-only-after-prompting"): + if heading.lower() not in attempt_markdown.lower(): + warnings.append(f"attempt_markdown has no `{heading}` section.") + + out = f"Attempt saved to {path}.\nNext: save_hld_diff against the frozen reference." + if warnings: + out += "\n\nWARNING: " + " ".join(warnings) + return out + + +@mcp.tool() +def save_hld_diff( + attempt_folder: str, + matched: str, + missed: str, + diverged: str, + diff_markdown: str, +) -> str: + """Compare the candidate's attempt against the frozen reference, as + diff.md. Call this after save_hld_attempt and before save_hld_evaluation — + the diff is what the scores are then justified by. + + Three buckets, and the distinction between the last two is the whole point: + matched — present in both reference and attempt. + missed — a reference INVARIANT absent from the attempt. A real gap. + diverged — the attempt took a different option at a CHOICE POINT. Not a + gap if the trade-off reasoning was sound; record the reasoning + they actually gave, so this can be re-read later. + + diff_markdown should carry a per-section table AND, separately, a + flow-level diff comparing the end-to-end traces. Two designs can list + identical boxes and still route a request completely differently — a + component-level diff alone would call that a match. + + Errors if reference.md or attempt.md is missing: a diff with only one side + is meaningless. + + Args: + attempt_folder: The folder returned by start_hld_mock_attempt. + matched: Semicolon-separated list. + missed: Semicolon-separated list of missed invariants. + diverged: Semicolon-separated list of choice-point deviations. + diff_markdown: The full written comparison, authored by you. + """ + folder = _resolve_hld_folder(attempt_folder) + if not folder: + return f"attempt_folder must name a folder inside {HLD_MOCK_DIR}." + for required in ("reference.md", "attempt.md"): + if not (folder / required).exists(): + return ( + f"No {required} in {folder} — a diff needs both sides to mean " + "anything. Run start_hld_mock_attempt / save_hld_attempt first." + ) + + meta = _hld_folder_meta(folder) + buckets = { + name: [x.strip() for x in raw.split(";") if x.strip()] + for name, raw in (("matched", matched), ("missed", missed), ("diverged", diverged)) + } + + lines = [ + f"# {meta['title'] or meta['problem_id']} — reference vs. attempt", + "", + f"_Diffed: {date.today().isoformat()} · id: `{meta['problem_id']}` · folder: `{folder.name}`_", + "", + *_markdown_table( + ["Bucket", "Count", "Meaning"], + [ + ("matched", len(buckets["matched"]), "in both reference and attempt"), + ("missed", len(buckets["missed"]), "**reference invariant absent from the attempt — a real gap**"), + ("diverged", len(buckets["diverged"]), "different option at a choice point — not a gap if the reasoning held"), + ], + ), + "", + ] + for name, heading in ( + ("matched", "## Matched"), + ("missed", "## Missed invariants"), + ("diverged", "## Diverged at choice points"), + ): + lines += [heading, ""] + lines += [f"- {item}" for item in buckets[name]] or ["- (none)"] + lines += [""] + lines += ["---", "", diff_markdown.strip(), ""] + + path = folder / "diff.md" + path.write_text("\n".join(lines), encoding="utf-8") + + out = ( + f"Diff saved to {path} " + f"({len(buckets['matched'])} matched, {len(buckets['missed'])} missed, " + f"{len(buckets['diverged'])} diverged).\n" + "Next: save_hld_evaluation." + ) + if "flow" not in diff_markdown.lower(): + out += ( + "\n\nWARNING: diff_markdown never mentions flows. Add a flow-level " + "diff comparing the end-to-end traces — identical component lists " + "can still route a request completely differently." + ) + return out + + +@mcp.tool() +def save_hld_evaluation( + attempt_folder: str, + verdict: str, + level_verdict: str, + rubric_scores: dict, + strengths: str, + gaps: str, + evaluation_markdown: str, + persona: str = "", +) -> str: + """Score an HLD mock as the interviewer. Writes evaluation.md into the + attempt folder, records the rubric scores so they aggregate across + sessions, and logs the session — so this REPLACES a separate log_session + call for HLD mocks. + + Call it last in the mock sequence, after save_hld_diff, so the scores are + justified by a written diff rather than by end-of-session recall. Score + honestly against the bar: inflated scores make the whole loop useless, + since these averages are what get_hld_feedback feeds into the next + session's problem choice. + + rubric_scores keys MUST come from this fixed list (1-5 each; omit a + dimension the session didn't exercise): + requirements — functional/non-functional, what they pinned down + capacity-estimation — did they justify the numbers themselves + architecture — component design, and whether each box is warranted + deep-dives — depth on the hard parts under drilling + scale-calibration — read the stated numbers vs. pattern-matched to FAANG + communication — structure, signposting, driving the session + composure — behaviour under pushback and persona pressure + Free-form keys are rejected: the aggregate is the whole value, and keys + that vary per session never aggregate. + + Args: + attempt_folder: The folder returned by start_hld_mock_attempt. + verdict: One of Strong Hire, Hire, Lean Hire, Lean No Hire, No Hire. + level_verdict: The verdict at level, e.g. "Hire@Senior, No-hire@Staff". + rubric_scores: dict of dimension -> integer 1-5, keys from the list above. + strengths: Semicolon-separated list of what they did well. + gaps: Semicolon-separated short weak-area tags. Prefer reusing rubric + dimension names so they aggregate with the scores. + evaluation_markdown: Your full written critique, authored by you. + persona: The interviewer persona used this session, revealed at the end + (standard / adversarial / silent / derailer). + """ + folder = _resolve_hld_folder(attempt_folder) + if not folder: + return f"attempt_folder must name a folder inside {HLD_MOCK_DIR}." + if not folder.is_dir(): + return f"No attempt folder at {folder}. Use list_hld_mock_attempts to see what exists." + + verdict = verdict.strip() + if verdict not in VALID_VERDICTS: + return f"Invalid verdict '{verdict}'. Use one of: {sorted(VALID_VERDICTS)}" + + unknown = [k for k in rubric_scores if k.strip().lower() not in HLD_RUBRIC] + if unknown: + return ( + f"Unknown rubric dimension(s): {unknown}. Scores only aggregate if " + f"keys come from the fixed vocabulary: {HLD_RUBRIC}" + ) + if not rubric_scores: + return f"rubric_scores is empty — score at least one dimension from {HLD_RUBRIC}." + clean_scores = {} + for area, score in rubric_scores.items(): + if not isinstance(score, int) or not (MIN_COMPETENCY_SCORE <= score <= MAX_COMPETENCY_SCORE): + return ( + f"Invalid score for '{area}': {score!r}. Scores must be " + f"integers {MIN_COMPETENCY_SCORE}-{MAX_COMPETENCY_SCORE}." + ) + clean_scores[area.strip().lower()] = score + + meta = _hld_folder_meta(folder) + slug = meta["problem_id"] + catalog_entry = {c["id"]: c for c in _load_catalog()["HLD"]}.get(slug) + display_title = meta["title"] or (catalog_entry["title"] if catalog_entry else slug) + today = date.today().isoformat() + average = sum(clean_scores.values()) / len(clean_scores) + + # Rendered in HLD_RUBRIC order (not dict order) so every evaluation.md + # reads the same way regardless of how the scores were passed in. + score_table = _markdown_table( + ["Dimension", "Score"], + [(dim, f"{clean_scores[dim]}/5") for dim in HLD_RUBRIC if dim in clean_scores] + + [("**average**", f"**{average:.1f}/5**")], + ) + + path = folder / "evaluation.md" + header = ( + f"# {display_title} — evaluation\n\n" + f"_Graded: {today} · Type: HLD · id: `{slug}` · verdict: **{verdict}**_\n\n" + f"**At level:** {level_verdict.strip() or '(not stated)'}\n\n" + ) + if persona.strip(): + header += f"**Interviewer persona:** {persona.strip()}\n\n" + path.write_text( + header + "\n".join(score_table) + "\n\n---\n\n" + evaluation_markdown.strip() + "\n", + encoding="utf-8", + ) + + # Rubric scores go through log_session's competency machinery under an + # "hld:" prefix, so they aggregate exactly like the LLD rubric without + # sharing its namespace. log_session also appends to revision.md and + # updates weak_areas + the per-problem tracker -- and it preserves any + # doc_path already recorded, which is what keeps this attempt folder out + # of list_practice_docs. Nothing here ever points a tracker doc_path at + # Mock Solutions/: the practice doc stays the revision artifact. + log_result = log_session( + topic=display_title, + interview_type="HLD", + verdict=verdict, + strengths=strengths, + gaps=gaps, + notes=( + f"HLD mock attempt in {folder.name}" + + (f" (persona: {persona.strip()})" if persona.strip() else "") + + f". At level: {level_verdict.strip()}. Evaluation: {path}." + ), + problem_id=slug, + competency_scores={f"{HLD_RUBRIC_PREFIX}{k}": v for k, v in clean_scores.items()}, + ) + + index = _load_index() + index.setdefault("hld_mock", []).append({ + "problem_id": slug, + "title": display_title, + "round": meta["round"], + "date": today, + "verdict": verdict, + "level_verdict": level_verdict.strip(), + "persona": persona.strip(), + "scores": clean_scores, + "average": average, + "folder": folder.name, + "folder_path": str(folder), + }) + _save_index(index) + + weakest = _weakest_dimensions(prefix=HLD_RUBRIC_PREFIX) + weak_note = ( + " Weakest dimensions now: " + + ", ".join(f"{d} ({avg:.1f})" for d, avg, _ in weakest) + + "." + ) if weakest else "" + + return ( + f"Evaluation saved to {path} (average {average:.1f}/5, verdict {verdict}" + + (f", {level_verdict.strip()}" if level_verdict.strip() else "") + + f").\n{log_result}{weak_note}\n" + "Next: save_practice_doc for the clean revision artifact — the attempt " + "folder keeps the evidence, warts intact." + ) + + +@mcp.tool() +def list_hld_mock_attempts(problem_id: str = "") -> str: + """List the HLD mock attempt folders under "Mock Solutions/": for each, + which of the four files exist (reference / attempt / diff / evaluation), + and the verdict if it's been graded. + + Call this at the start of an HLD session to avoid re-posing a problem the + candidate has already attempted, and to spot attempts left incomplete + (a reference with no attempt, an attempt never graded). Use + read_hld_mock_file to open any listed file. + + Args: + problem_id: Optional — limit to one problem's attempts. + """ + if not HLD_MOCK_DIR.exists(): + return ( + f"No HLD mock attempts yet — {HLD_MOCK_DIR} doesn't exist. " + "Use start_hld_mock_attempt to freeze the first reference." + ) + + graded = {rec.get("folder"): rec for rec in _load_index().get("hld_mock", [])} + folders = sorted(p for p in HLD_MOCK_DIR.iterdir() if p.is_dir() and not p.name.startswith(".")) + if problem_id.strip(): + slug = _slugify(problem_id) + folders = [f for f in folders if _hld_folder_meta(f)["problem_id"] == slug] + if not folders: + return f"No HLD mock attempts for `{slug}` under {HLD_MOCK_DIR}." + if not folders: + return f"No HLD mock attempts found under {HLD_MOCK_DIR}. Use start_hld_mock_attempt to open one." + + rows = [] + incomplete = [] + for folder in folders: + meta = _hld_folder_meta(folder) + rec = graded.get(folder.name) + present = {f: (folder / f).exists() for f in HLD_MOCK_FILES} + rows.append(( + f"`{folder.name}`", + f"`{meta['problem_id']}`", + meta["date"], + *("yes" if present[f] else "—" for f in HLD_MOCK_FILES), + rec["verdict"] if rec else "not graded", + )) + if not all(present.values()): + missing = [f for f in HLD_MOCK_FILES if not present[f]] + incomplete.append(f"{folder.name} (missing {', '.join(missing)})") + + out = [f"HLD mock attempts under {HLD_MOCK_DIR}:", ""] + out += _markdown_table( + ["Folder", "Problem", "Date", "Reference", "Attempt", "Diff", "Evaluation", "Verdict"], + rows, + ) + if incomplete: + out += ["", "Incomplete: " + "; ".join(incomplete)] + return "\n".join(out) + + +@mcp.tool() +def read_hld_mock_file(attempt_folder: str, filename: str) -> str: + """Read one file from an HLD mock attempt folder in full — e.g. to re-read + the frozen reference before writing the diff, or to pull up a past + attempt's evaluation before a re-attempt. + + Args: + attempt_folder: The folder name (or path) from list_hld_mock_attempts. + filename: One of reference.md, attempt.md, diff.md, evaluation.md. + """ + if filename.strip() not in HLD_MOCK_FILES: + return f"filename must be one of {list(HLD_MOCK_FILES)}." + folder = _resolve_hld_folder(attempt_folder) + if not folder: + return f"attempt_folder must name a folder inside {HLD_MOCK_DIR}." + path = folder / filename.strip() + if not path.exists(): + return f"No {filename.strip()} in {folder}. Use list_hld_mock_attempts to see which files exist." + return path.read_text(encoding="utf-8") + + +@mcp.tool() +def get_hld_feedback() -> str: + """Summarize how the user is performing across HLD mock interviews: rubric + averages per dimension, the weakest dimensions, and recent verdict history. + + Call this at the START of an HLD session, alongside get_catalog, and let it + drive the session: pick a problem that forces the weakest dimensions, and + push hardest on them during the interview. Reading it afterwards records + scores; reading it beforehand is what makes the loop tune itself. + """ + records = _load_index().get("hld_mock", []) + if not records: + return ( + "No HLD mock evaluations recorded yet. Use start_hld_mock_attempt " + "to freeze a reference, then save_hld_attempt / save_hld_diff / " + "save_hld_evaluation once the session is done — the rubric " + "averages build up from there." + ) + + lines = [ + f"{len(records)} HLD mock attempt(s) evaluated.", + "", + "Rubric averages (1-5):", + *(f"- {d}: {avg:.1f} ({n} rated)" for d, avg, n in _rated_dimensions(HLD_RUBRIC_PREFIX)), + ] + unrated = _unrated_dimensions(HLD_RUBRIC, HLD_RUBRIC_PREFIX) + if unrated: + lines.append(f"- not yet exercised: {', '.join(unrated)}") + + weakest = _weakest_dimensions(prefix=HLD_RUBRIC_PREFIX) + if weakest: + lines += [ + "", + "Weakest dimensions — bias the next problem toward these, and " + "probe them hard during the interview:", + *(f"- {d} ({avg:.1f}/5 over {n} attempt(s))" for d, avg, n in weakest), + ] + + lines += ["", "Recent attempts:", ""] + lines += _markdown_table( + ["Date", "Problem", "Round", "Verdict", "Average", "Weakest this round"], + _mock_history_rows(records, limit=10), + ) + levels = [f"{r['date']} {r['problem_id']}: {r['level_verdict']}" for r in records[-5:] if r.get("level_verdict")] + if levels: + lines += ["", "Recent verdicts at level:", *(f"- {l}" for l in levels)] + lines += ["", f"Attempt folders: {HLD_MOCK_DIR}"] + return "\n".join(lines) + + if __name__ == "__main__": mcp.run(transport="stdio") diff --git a/skills/hld-interviewer/SKILL.md b/skills/hld-interviewer/SKILL.md new file mode 100644 index 0000000..42cf4d7 --- /dev/null +++ b/skills/hld-interviewer/SKILL.md @@ -0,0 +1,337 @@ +--- +name: hld-interviewer +description: Run a realistic, Staff-level mock High-Level Design (HLD) system-design interview where Claude acts as the interviewer and the user is the candidate. Use this whenever the user wants to practice HLD, run a mock system-design interview, be quizzed on designing a system (e.g. rate limiter, URL shortener, live streaming, chat, payments), or rehearse for a Staff/Senior design round. Trigger even if they just say "let's do a mock HLD" or name a system to design under interview conditions. At the end, always produce consolidated feedback and persist the design as a revision doc via the interview-memory MCP. +--- + +# HLD Interviewer + +Act as a Staff-level system-design interviewer, modeled on the +HelloInterview / "Jordan has no life" bar. The candidate leads the design; +you probe, pressure-test, and score. You are NOT a co-designer and NOT a +tutor mid-interview — you are the interviewer. + +## Session setup + +- **One system per conversation.** Keep context focused; start a fresh + chat for the next problem. +- **Call `get_hld_feedback` first**, alongside `get_catalog`. It returns + the rubric averages across past mocks, the weakest dimensions, and + recent verdicts at level. Let it drive the pick and the in-session + pressure: choose a problem that *forces* the weak dimensions and push + hardest there. Choosing blind wastes the round. +- Also call `list_hld_mock_attempts` to see what's already been + attempted (and what was left ungraded), so you don't re-pose a problem + they worked last month. +- If the user has no preference, either pull their catalog and pick a + problem that's **due for revision**, or offer a fresh Staff-bar problem + they haven't logged. Use `get_catalog` / `list_practice_docs` for this. +- **Vary the scale.** Roughly 1 in 3 sessions, deliberately pick or frame + a LOW-scale problem (internal tool, B2B app, thousands of users, tens + of QPS). Do not hint that it's a low-scale round — part of the test is + whether the candidate reads the numbers instead of pattern-matching to + FAANG-scale designs. +- **Prefer variants over canonicals.** Roughly half the time, pose a + *variant* of a canonical system instead of the canonical itself. Take + a system from the catalog and shift exactly one axis: + - **Actor model** — 1-1 chat → customer ↔ agent-pool support chat; + social feed → moderated marketplace listings. + - **Scale** — Twitter → internal company feed; YouTube → corporate + training video portal. + - **Constraint** — URL shortener → vanity URLs with expiry + + analytics; ride sharing → scheduled-only rides. + - **Domain** — Uber → ambulance dispatch (latency is life-critical, + supply is tiny); Ticketmaster → vaccine appointment booking. + Real interviews increasingly ask variants precisely to defeat + memorized designs; the signal is whether the candidate adapts. +- Confirm nothing else — get into the interview quickly. + +## Reference solution (pre-commit — do this BEFORE the first question) + +Before posing the problem, silently establish the grading reference: + +1. **Always generate the reference fresh yourself, now** — before the + candidate has said anything about the design. Do not adopt an old + revision doc as the reference, even if one exists (past docs may + contain errors accepted under candidate pushback). Rules for + generation: + - Stick to standard, widely-documented patterns — what is commonly + used in production and commonly presented at the Staff bar. No + exotic or niche tech. + - Mark any claim you are not certain of with `[verify]`. Never + grade the candidate against a `[verify]`-marked claim. +2. If a revision doc exists for this problem (`list_practice_docs`), + read it *after* generating your reference, as a cross-check only: + adopt from it anything that improves your reference AND that you + independently agree is standard; diff the rest and surface + discrepancies (possible sycophancy artifacts from past sessions) in + end-of-session feedback so the doc gets corrected. +3. Structure the reference as: + - **Invariants** — what every acceptable solution must have. + - **Choice points** — dimensions where multiple designs are fine; + for each, the 2–3 standard options and their trade-offs. Score + whether the candidate knows the trade-offs, not whether they pick + your favorite. + - **Scale verdict** — from the stated traffic: which components the + numbers justify, and which would be over-engineering. + - **Delta table (variants only)** — vs. the nearest canonical + system: what carries over unchanged, what breaks, what's new. + (e.g., support chat vs. WhatsApp: WebSocket delivery and message + storage carry over; the peer model breaks — needs a + routing/assignment engine; new pieces: agent workload balancing, + conversation lifecycle state machine, transcript/CRM export.) + Grading a variant centers on whether the candidate identifies + these deltas. +4. The reference must include an **`## End-to-end flows`** section: + at least three numbered, sequential flows — the write path, the + primary read-or-execute path, and a failure/recovery path — each step + naming the component, the operation and the datastore it touches + ("sweeper runs `SELECT ... FOR UPDATE SKIP LOCKED` on `job_runs` + where `next_fire_time < now + 5min`", not "sweeper picks up jobs"). + The end-of-session diff compares flows, not just which components got + named — two designs can list identical boxes and still route a + request completely differently. +5. **Persist it now, before posing the problem.** Call + `start_hld_mock_attempt(problem_id, title, reference_markdown, + variant_of=..., difficulty=...)`. It writes `reference.md` into a + dated attempt folder and returns that folder path — hold onto it, the + three end-of-session tools all take it. +6. The reference is **frozen** for the rest of the session. Every + judgment you make compares the candidate's answer to it. You may not + revise it mid-interview — and now you cannot: `start_hld_mock_attempt` + refuses to overwrite an existing `reference.md`, and the file's + `written_at` comes from the server's clock, so the pre-commitment is + enforced rather than merely promised. Do not call the tool again for + this session. + +## Interview flow (drive it; don't lecture) + +1. Give the one-line prompt, then **stop**. Let the candidate scope it. +2. **Requirements** — functional + non-functional. If they skip scale, + consistency, availability, or latency targets, prompt **once** + ("anything else you want to pin down?"). If they still skip it, let + them proceed — a real interviewer would — and let the miss bite + later in the design. Trace the consequence back to the missed + requirement in feedback. +3. **Capacity estimation** — make them justify the numbers, not you. + Their numbers become binding: later components must be consistent + with them. +4. **High-level design** — candidate draws (ASCII/text). You ask "why?", + "what breaks at 10x?", "where's the bottleneck?" +5. **Deep dives** — pick the 1–2 hardest parts and drill hard. +6. **Trade-offs** — bottlenecks, failure modes, what changes under + different constraints. + +**Phase budget:** core architecture and deep dives are the bulk of the +session. Edge cases and failure modes get at most ~15% of turns, and +never before the core architecture is complete. **Max 2 follow-ups per +edge case**, then move on — do not rabbit-hole. + +## Pacing (time pressure) + +Simulate a 40-minute round using turn count as the clock: +**~24 candidate turns total** (≈ requirements 4, estimation 3, +high-level 8, deep dives 6, trade-offs/twist 3). + +- Announce checkpoints at 25% / 50% / 75%: "We're at the halfway mark — + you're still in requirements." State it neutrally; do not extend the + budget to compensate. +- **Hard stop at the budget.** If the candidate hasn't reached deep + dives, the interview still ends. Running out of time is itself a + scored failure — do not quietly grant extra turns. +- If the candidate stalls on one point for 3+ turns, do what a real + interviewer does: "In the interest of time, let's move on." + +## Interviewer persona (pick one per session, reveal only at the end) + +Randomly adopt one persona at session start. Keep grading identical — +the frozen reference and pushback protocol always apply; only the +*conversational style* changes. + +- **Standard** — engaged, neutral (default weight ~40%). +- **Adversarial** — challenges frequently, including **1–2 challenges + on answers that are actually correct** ("Are you sure that holds + under a partition?"). Purpose: train composure. A candidate who + calmly defends a correct answer with reasoning scores UP; one who + abandons a correct answer under pressure gets this flagged + prominently in feedback. +- **Silent** — minimal acknowledgments, no encouragement, one-word + bridges. Tests whether the candidate keeps structure without social + feedback. +- **Derailer** — occasionally interjects a tangent or premature edge + case; the candidate should park it politely ("I'll cover that in the + deep dive") and hold the thread. + +In feedback, reveal the persona and rate **composure**: how the +candidate handled pressure, silence, or derailment. + +## Interviewer rules + +- **ONE question per turn.** Never restate the full design back to them. +- Don't hand out the answer. Nudge with a question, not a solution. +- **Immediate failure-mode probe:** whenever the candidate commits to a + major component or pattern (queue, cache, shard scheme, leader, + fan-out...), your next question is "what's the failure mode of that + choice?" — asked at decision time, not saved for the deep dive. Skip + it only if they pre-empted it themselves (that's a plus; note it). +- **Justify every box:** if a component isn't warranted by the + candidate's own stated numbers, challenge it: "What number in your + estimation requires Kafka here?" Over-engineering is a real gap, not + a bonus. +- **Map-and-diverge probe (variants):** early in the design phase, ask + "How is this similar to and different from [the canonical system]?" + A Staff candidate articulates the mapping themselves. Copying the + canonical design wholesale without addressing the deltas is flagged + the same way over-engineering is. +- **One controlled twist per session:** after the core architecture is + scored (never before), introduce exactly one requirement change — + "Product now wants X" (e.g., file attachments, conversation transfer + between agents, a bot triage layer). Score whether the design extends + calmly or requires a rewrite. One twist only; it must not eat the + phase budget. +- Push back on hand-waving: "How does that stay consistent under a + partition?" +- Stay silent on scoring until the end. Track gaps internally; don't + narrate them mid-interview. +- Keep your turns short. You're an interviewer, not a lecturer. + +## Technical honesty (anti-sycophancy) + +- Do **not** validate a design because the candidate sounds confident. + Correctness is judged against the frozen reference and the candidate's + own numbers, not their tone. +- **Pushback protocol:** if the candidate disagrees with your + assessment, do not re-evaluate. Ask them to justify. A **new technical + argument** can change your judgment; **persistence, confidence, or + repetition cannot**. Changing an assessment because the candidate + pushed harder is a failure of this skill. +- If a proposal doesn't work, say so plainly and name the failure: + "That loses writes under a partition." / "That's O(n) per request — + won't hold at your stated QPS." Don't soften a real flaw into a vague + nudge. +- **"Sounds good, let's continue" is banned when it isn't good.** Never + agree to move on until a genuine gap is actually resolved. +- If the candidate is right, confirm briefly and push deeper — don't + praise. If they're wrong, don't supply the fix; make them find it. +- Distinguish an *acceptable trade-off* from *broken*. Flag broken as + broken. A deviation from the reference at a **choice point** with + sound trade-off reasoning is NOT a gap; a missed **invariant** is. + +## No hallucination + +- Don't invent numbers, benchmarks, or claim a specific DB/tool behaves a + certain way unless you're sure. If unsure, say "verify that" rather than + asserting it as fact. +- Don't declare a design "passes" or "fails" a scale target without the + candidate's own estimation backing it — make them show the math. +- If the candidate cites a fact you can't confirm, don't rubber-stamp it; + ask them to justify it. + +## End of interview + +1. Give **consolidated feedback** in chat: strengths, Staff-bar gaps, and + what to revise. Be honest — a mock is worthless if it flatters. + **Open with a verdict at level:** "Hire / No-hire at Senior; Hire / + No-hire at Staff — and the ONE thing separating you from the next + level is X." No hedged verdicts. + Always include these sections: + - **Session scorecard (1–5 each):** requirements, capacity + estimation, architecture, deep dives, scale calibration, + communication (structure, signposting, driving the session), and + composure (behavior under pushback/persona pressure). Same seven + dimensions every session, so scores are comparable over time. + - **Missed invariants** vs. **choice-point deviations** (only the + former are real gaps). + - **Choices that came back to bite:** each decision that later caused + trouble, and the question that would have caught it at decision + time. + - **Scale calibration:** components not justified by the stated + numbers (over-engineering), or components missing that the numbers + demanded (under-engineering). Cite the number in each case. + - **Pattern-matching vs. adaptation (variants):** where the + candidate correctly adapted the canonical pattern, where they + imported machinery the variant didn't need, and which deltas they + missed. +2. Then persist the session's **evidence**, in this order, all three + taking the attempt folder `start_hld_mock_attempt` returned: + + a. **`save_hld_attempt(attempt_folder, attempt_markdown, raw_turns)`** + — what the candidate actually designed. `raw_turns` matters more + than `attempt_markdown`: pass their design turns close to + verbatim. Writing this from memory instead tidies the reasoning + unconsciously — filling in a justification they never gave, + straightening out an explanation that doubled back — and that + silently destroys the diff. `attempt_markdown` supplies two + sections: `## Interviewer's structured rendering` (the same design + normalised for mechanical comparison) and + `## Arrived-at-only-after-prompting` (every conclusion they + reached only AFTER you named the gap, with your prompting question + quoted). Do not correct the attempt toward the reference. + + b. **`save_hld_diff(attempt_folder, matched, missed, diverged, + diff_markdown)`** — `missed` is a reference **invariant** absent + from the attempt (a real gap); `diverged` is a different option at + a **choice point** (not a gap if the trade-off reasoning was + sound — record the reasoning they gave). `diff_markdown` carries a + per-section table AND a separate flow-level diff of the end-to-end + traces. + + c. **`save_hld_evaluation(attempt_folder, verdict, level_verdict, + rubric_scores, strengths, gaps, evaluation_markdown, persona)`** — + the scorecard, with `rubric_scores` keyed by exactly these seven: + `requirements`, `capacity-estimation`, `architecture`, + `deep-dives`, `scale-calibration`, `communication`, `composure`. + Free-form keys are rejected. **This logs the session too — do not + call `log_session` separately for an HLD mock.** + +3. Finally call **`save_practice_doc`** (`problem_type=HLD`) with a full + write-up authored by you. If the problem matches a catalog entry, pass + its `problem_id`; the doc overwrites any existing one for that problem. + + The write-up should contain: + - Requirements (functional + non-functional) + - Capacity estimation + - High-level architecture (components as an ASCII diagram) + - **`## End-to-end flows`** — immediately after the diagram and + before the deep dives, same rules as the reference: minimum three + numbered flows (write path, primary read-or-execute path, + failure/recovery path), each step naming the component, the + operation and the datastore touched, and noting what recovers a + step that can fail. The diagram shows what exists; this shows what + happens, which is what's missing when you reread a doc six weeks + later. + - API design + - Data model + - Deep dives on the hard parts + - Trade-offs and what you'd change under different constraints + - Invariants vs. choice points for this system, **taken from your + fresh pre-committed reference** (corrected for anything the + session disproved) — so every save re-audits the doc and the + revision material stays clean. + - For variants: save under the **variant's own name** (do NOT + overwrite the canonical's doc), include the delta table, and + cross-reference the canonical problem — the catalog should grow + families (chat → WhatsApp, support chat, Discord rooms), not + isolated docs. + - **Append a `## Session log` section** at the end of the doc with: + date, persona used, verdict (level), and the seven-dimension + scorecard as one line, e.g. + `2026-07-15 | adversarial | Hire@Senior, No-hire@Staff | req:4 est:3 arch:4 deep:3 scale:4 comm:3 composure:2`. + Keep prior log lines when overwriting a doc — the log accumulates + across revisions so trends per dimension can be pulled by reading + the logs across all docs. + +**The practice doc is the clean revision artifact; the attempt folder is +the record of what happened.** Never rewrite `attempt.md` toward the +correct answer — an attempt groomed into the reference reads clean six +weeks later and tells the candidate nothing about which parts they +actually got right. Correct material belongs in `reference.md` and in +the practice doc. Existing HLD practice docs may be *augmented* with an +`## End-to-end flows` section on next revisit, since that describes the +correct design and overwrites nothing the candidate produced. + +## Token discipline + +- One system per conversation. +- ASCII over rendered diagrams. +- No feedback until the end; short interviewer turns throughout. +- Keep web search off — not needed for HLD. diff --git a/test_hld_tools.py b/test_hld_tools.py new file mode 100644 index 0000000..25d236b --- /dev/null +++ b/test_hld_tools.py @@ -0,0 +1,346 @@ +""" +Hermetic tests for the HLD mock loop: the pre-committed reference, the +attempt, the diff and the seven-dimension evaluation. + +Builds a synthetic HLD_SOLUTIONS_DIR in a temp directory and exercises every +HLD tool against it. Nothing outside the temp dirs is read or written, so this +is safe to run at any time -- in particular it never touches the real +HLD_SOLUTIONS_DIR the user keeps their designs in. + +The acceptance checklist for this feature is the test plan, one named check +each: reference.md can't be overwritten, written_at comes from the server +clock, a diff without both sides errors, a free-form rubric key is rejected, +Mock Solutions/ never leaks into list_practice_docs, a doc with no +"## End-to-end flows" warns but still writes, and get_hld_feedback on an +empty history returns cleanly. + +Usage: + python3 test_hld_tools.py +""" + +import importlib +import os +import sys +import tempfile +from datetime import date, datetime +from pathlib import Path + +FAILS = [] + + +def check(label, cond, detail=""): + print(f"{'PASS' if cond else 'FAIL'} {label}" + (f" [{detail}]" if detail and not cond else "")) + if not cond: + FAILS.append(label) + + +FLOWS = """ +## End-to-end flows + +### Flow 1: job registration +1. API writes to `jobs` via `INSERT`. + +### Flow 2: firing +1. Sweeper runs `SELECT ... FOR UPDATE SKIP LOCKED` on `job_runs`. + +### Flow 3: worker crash +1. Lease expires; sweeper re-claims the run. +""" + +REFERENCE = "## Invariants\n\n- exactly-once firing\n" + FLOWS +ATTEMPT = ( + "## Interviewer's structured rendering\n\n- polling sweeper\n" + "\n## Arrived-at-only-after-prompting\n\n- idempotency keys, after " + '"what happens if the worker dies mid-run?"\n' +) + + +def main() -> int: + tmp = Path(tempfile.mkdtemp(prefix="hld-tools-test-")) + hld = tmp / "HLD" + hld.mkdir(parents=True) + + # Point every root at the sandbox BEFORE importing server: the module reads + # these at import time, so a real root would be created and written to. + os.environ["INTERVIEW_PREP_DIR"] = str(tmp / "prep") + os.environ["HLD_SOLUTIONS_DIR"] = str(hld) + os.environ["LLD_SOLUTIONS_DIR"] = str(tmp / "LLD") + os.environ["DSA_SOLUTIONS_DIR"] = str(tmp / "dsa") + os.environ["BEHAVIORAL_SOLUTIONS_DIR"] = str(tmp / "behavioral") + sys.path.insert(0, str(Path(__file__).resolve().parent)) + server = importlib.import_module("server") + server = importlib.reload(server) + check("test sandbox is isolated from the real HLD root", + server.HLD_SOLUTIONS_DIR == hld, str(server.HLD_SOLUTIONS_DIR)) + + today = date.today().isoformat() + slug = "design-distributed-task-scheduler" + + # --- empty history ----------------------------------------------------- + print("\n-- get_hld_feedback on an empty history --") + empty = server.get_hld_feedback() + check("get_hld_feedback on empty history returns cleanly", + "No HLD mock evaluations recorded yet" in empty, empty[:120]) + check("list_hld_mock_attempts before any attempt returns cleanly", + "doesn't exist" in server.list_hld_mock_attempts()) + + # --- start_hld_mock_attempt ------------------------------------------- + print("\n-- start_hld_mock_attempt --") + started = server.start_hld_mock_attempt( + problem_id=slug, + title="Design a Distributed Task Scheduler", + reference_markdown=REFERENCE, + variant_of="design-cron", + difficulty="Hard", + ) + folder = hld / "Mock Solutions" / f"{today}-{slug}" + reference = folder / "reference.md" + check("attempt folder created with the dated name", folder.is_dir(), str(folder)) + check("folder path returned to the caller", str(folder) in started, started[:200]) + check("reference.md written", reference.exists()) + + ref_text = reference.read_text() + check("frontmatter records the problem id", f"problem_id: {slug}" in ref_text) + check("frontmatter records frozen: true", "frozen: true" in ref_text) + check("frontmatter records variant_of", "variant_of: design-cron" in ref_text) + check("reference body preserved", "exactly-once firing" in ref_text) + + stamp = next((l[len("written_at: "):] for l in ref_text.splitlines() + if l.startswith("written_at: ")), "") + parsed = None + try: + parsed = datetime.fromisoformat(stamp) + except ValueError: + pass + check("written_at parses as a timestamp", parsed is not None, stamp) + # The server clock, not a tool argument -- there is no parameter that could + # backdate it, and the value must agree with this machine's own clock. + check("written_at comes from the server clock (today, not caller-supplied)", + parsed is not None and parsed.date() == date.today(), stamp) + check("no tool argument can set written_at", + "written_at" not in server.start_hld_mock_attempt.__doc__) + + # --- refusing to overwrite a frozen reference -------------------------- + print("\n-- the pre-commitment is binding --") + again = server.start_hld_mock_attempt( + problem_id=slug, title="Design a Distributed Task Scheduler", + reference_markdown="## Invariants\n\n- rewritten after the fact\n" + FLOWS, + ) + check("second start_hld_mock_attempt on the same folder errors", + "refusing to overwrite" in again.lower(), again[:160]) + check("the error names the existing folder", folder.name in again, again[:200]) + check("the error points at the -r2 escape hatch", "round_no=2" in again, again[:240]) + check("the frozen reference is untouched", + "exactly-once firing" in reference.read_text() + and "rewritten after the fact" not in reference.read_text()) + + r2 = server.start_hld_mock_attempt( + problem_id=slug, title="Design a Distributed Task Scheduler", + reference_markdown=REFERENCE, round_no=2, + ) + r2_folder = hld / "Mock Solutions" / f"{today}-{slug}-r2" + check("round_no=2 opens a separate folder alongside", r2_folder.is_dir(), r2[:160]) + + # --- the flows warning ------------------------------------------------- + print("\n-- the End-to-end flows warning --") + no_flows = server.start_hld_mock_attempt( + problem_id="design-rate-limiter", title="Design a Rate Limiter", + reference_markdown="## Invariants\n\n- token bucket\n", + ) + rl_folder = hld / "Mock Solutions" / f"{today}-design-rate-limiter" + check("reference with no flows section still writes", + (rl_folder / "reference.md").exists()) + check("reference with no flows section warns", + "End-to-end flows" in no_flows and "WARNING" in no_flows, no_flows[:200]) + check("reference WITH a flows section does not warn", "WARNING" not in started, started[:200]) + + # --- path guards ------------------------------------------------------- + print("\n-- path guards --") + escape = server.save_hld_attempt("../../../etc", ATTEMPT, raw_turns="x") + check("a traversing attempt_folder is refused", "must name a folder inside" in escape, escape[:120]) + check("an absolute path outside the mock dir is refused", + "must name a folder inside" in server.save_hld_attempt(str(tmp), ATTEMPT), escape[:120]) + check("the mock root itself is not an attempt folder", + server._resolve_hld_folder(str(hld / "Mock Solutions")) is None) + check("a folder name inside the mock dir resolves", + server._resolve_hld_folder(folder.name) == folder.resolve()) + + # --- diff before attempt errors --------------------------------------- + print("\n-- save_hld_diff needs both sides --") + early = server.save_hld_diff(folder.name, "a", "b", "c", "flow diff") + check("save_hld_diff with a missing attempt.md errors", + "No attempt.md" in early, early[:160]) + check("no diff.md written when a side is missing", not (folder / "diff.md").exists()) + + # --- save_hld_attempt -------------------------------------------------- + print("\n-- save_hld_attempt --") + saved = server.save_hld_attempt( + folder.name, ATTEMPT, + raw_turns="so I'd have a table of jobs, and, um, a poller that reads it", + ) + attempt = folder / "attempt.md" + text = attempt.read_text() + check("attempt.md written", attempt.exists(), saved[:120]) + check("verbatim turns kept under 'As stated by the candidate'", + "## As stated by the candidate" in text and "um, a poller" in text) + check("structured rendering section present", + "## Interviewer's structured rendering" in text) + check("arrived-at-only-after-prompting section present", + "## Arrived-at-only-after-prompting" in text) + check("no warning when raw_turns and both sections are supplied", + "WARNING" not in saved, saved[:200]) + + bare = server.save_hld_attempt(r2_folder.name, "## Some notes\n\njust prose\n") + check("missing raw_turns warns that the diff grades a reconstruction", + "raw_turns was empty" in bare, bare[:240]) + check("missing arrived-at-only-after-prompting section warns", + "Arrived-at-only-after-prompting" in bare, bare[:300]) + check("the attempt still writes despite the warnings", + (r2_folder / "attempt.md").exists()) + + # --- save_hld_diff ----------------------------------------------------- + print("\n-- save_hld_diff --") + diffed = server.save_hld_diff( + folder.name, + matched="job store; sweeper", + missed="exactly-once firing", + diverged="polling instead of a timer wheel", + diff_markdown="### Flow-level diff\n\nReference fires via lease; attempt polls.\n", + ) + diff_md = (folder / "diff.md").read_text() + check("diff.md written", (folder / "diff.md").exists(), diffed[:120]) + check("missed invariants listed as a real gap", + "## Missed invariants" in diff_md and "exactly-once firing" in diff_md) + check("diverged kept separate from missed", + "## Diverged at choice points" in diff_md and "timer wheel" in diff_md) + check("bucket counts reported back", "1 matched" not in diffed and "2 matched" in diffed, diffed[:160]) + check("a diff that never mentions flows warns", + "WARNING" in server.save_hld_diff(r2_folder.name, "a", "b", "c", "component table only")) + check("the good diff.md is left intact", "timer wheel" in (folder / "diff.md").read_text()) + + # --- save_hld_evaluation ---------------------------------------------- + print("\n-- save_hld_evaluation --") + good_scores = {"requirements": 4, "capacity-estimation": 3, "architecture": 4, + "deep-dives": 3, "scale-calibration": 4, "communication": 3, + "composure": 2} + bad_key = server.save_hld_evaluation( + folder.name, "Hire", "Hire@Senior", {"requirements": 4, "vibes": 5}, + "s", "g", "body", + ) + check("a rubric key outside the fixed seven is rejected", + "Unknown rubric dimension" in bad_key and "vibes" in bad_key, bad_key[:160]) + check("nothing written when the rubric key is rejected", + not (folder / "evaluation.md").exists()) + check("an LLD rubric key is rejected on the HLD scorecard", + "Unknown rubric dimension" in server.save_hld_evaluation( + folder.name, "Hire", "x", {"class-decomposition": 4}, "s", "g", "b")) + check("an out-of-range score is rejected", + "Scores must be integers" in server.save_hld_evaluation( + folder.name, "Hire", "x", {"requirements": 9}, "s", "g", "b")) + check("an invalid verdict is rejected", + "Invalid verdict" in server.save_hld_evaluation( + folder.name, "Maybe", "x", good_scores, "s", "g", "b")) + + evaluated = server.save_hld_evaluation( + attempt_folder=folder.name, + verdict="Lean Hire", + level_verdict="Hire@Senior, No-hire@Staff", + rubric_scores=good_scores, + strengths="clear structure; drove the session", + gaps="capacity-estimation; composure", + evaluation_markdown="Numbers were asserted, not derived.", + persona="adversarial", + ) + eval_md = (folder / "evaluation.md").read_text() + check("evaluation.md written", (folder / "evaluation.md").exists(), evaluated[:160]) + check("verdict at level recorded", "Hire@Senior, No-hire@Staff" in eval_md) + check("persona recorded", "adversarial" in eval_md) + check("scorecard rendered in fixed rubric order", + eval_md.index("requirements") < eval_md.index("composure")) + check("session logged (no separate log_session needed)", + "Logged session" in evaluated, evaluated[:240]) + revision = (tmp / "prep" / "revision.md").read_text() + check("session appended to revision.md", "[HLD] Design a Distributed Task Scheduler" in revision) + + # --- aggregation ------------------------------------------------------- + print("\n-- aggregation --") + feedback = server.get_hld_feedback() + check("get_hld_feedback reports the attempt", "1 HLD mock attempt(s) evaluated" in feedback, feedback[:120]) + check("rubric averages reported", "composure: 2.0" in feedback, feedback[:400]) + check("weakest dimension surfaced first", + "composure (2.0/5" in feedback, feedback[:600]) + check("verdict at level surfaced", "Hire@Senior, No-hire@Staff" in feedback) + scores = server._load_index()["competency_scores"] + check("scores stored under the hld: prefix", "hld:composure" in scores, str(list(scores)[:5])) + check("HLD scores don't pollute the LLD rubric namespace", + not any(k.startswith("lld:") for k in scores), str(list(scores))) + check("HLD rubric slice reads back un-prefixed", + "composure" in server._rubric_scores(server.HLD_RUBRIC_PREFIX)) + check("LLD feedback unaffected by HLD scores", + "No LLD mock evaluations recorded yet" in server.get_lld_feedback()) + + summary = server.get_progress_summary() + check("progress summary reports the HLD rubric in its own block", + "HLD mock rubric" in summary, summary[-600:]) + check("HLD scores are not reported as behavioral competencies", + "Competency scores (behavioral" not in summary, summary[-600:]) + check("progress summary points at get_hld_feedback", + "Call get_hld_feedback before an HLD session" in summary, summary[-400:]) + + # --- listing ----------------------------------------------------------- + print("\n-- list_hld_mock_attempts --") + listing = server.list_hld_mock_attempts() + check("graded attempt shows its verdict", "Lean Hire" in listing, listing[:400]) + check("ungraded attempt shown as not graded", "not graded" in listing) + check("incomplete attempts called out", + "Incomplete:" in listing and f"{today}-design-rate-limiter" in listing, listing[:800]) + filtered = server.list_hld_mock_attempts(problem_id="design-rate-limiter") + check("filtering by problem id keeps the matching attempt", + f"`{today}-design-rate-limiter`" in filtered, filtered[:400]) + check("filtering by problem id excludes other problems", + f"`{today}-{slug}`" not in filtered, filtered[:400]) + check("unknown problem id reports cleanly", + "No HLD mock attempts for" in server.list_hld_mock_attempts(problem_id="nope")) + + # --- read_hld_mock_file ------------------------------------------------ + print("\n-- read_hld_mock_file --") + check("reference.md readable in full", + "exactly-once firing" in server.read_hld_mock_file(folder.name, "reference.md")) + check("a filename outside the four is refused", + "filename must be one of" in server.read_hld_mock_file(folder.name, "../../../etc/passwd")) + check("a missing file reports cleanly", + "No diff.md in" in server.read_hld_mock_file(rl_folder.name, "diff.md")) + + # --- practice docs stay separate from the evidence --------------------- + print("\n-- Mock Solutions/ vs. list_practice_docs --") + doc = server.save_practice_doc( + "HLD", "Design a Distributed Task Scheduler", + "## Requirements\n\n- fire jobs on time\n" + FLOWS, problem_id=slug, + ) + check("practice doc saved to the HLD root, not the mock folder", + str(hld / f"{slug}.md") in doc, doc[:200]) + docs = server.list_practice_docs("HLD") + check("list_practice_docs surfaces the practice doc", f"`{slug}`" in docs, docs[:400]) + check("list_practice_docs surfaces nothing under Mock Solutions/", + "Mock Solutions" not in docs, docs[:600]) + + no_flow_doc = server.save_practice_doc("HLD", "Design a Rate Limiter", "## Requirements\n\n- limit\n") + check("HLD doc without flows still writes", + (hld / "design-a-rate-limiter.md").exists()) + check("HLD doc without flows warns in the tool result", + "WARNING" in no_flow_doc and "End-to-end flows" in no_flow_doc, no_flow_doc[:240]) + check("HLD doc with flows does not warn", "WARNING" not in doc, doc[:200]) + lld_doc = server.save_practice_doc("LLD", "Design a Parking Lot", "## Classes\n\n- Lot\n") + check("the flows warning is HLD-only (no false alarm on LLD)", + "WARNING" not in lld_doc, lld_doc[:200]) + + print() + if FAILS: + print(f"{len(FAILS)} failure(s): " + ", ".join(FAILS)) + return 1 + print(f"All checks passed. (sandbox: {tmp})") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/test_server.py b/test_server.py index 5588455..da03f51 100644 --- a/test_server.py +++ b/test_server.py @@ -26,6 +26,9 @@ "start_mock_attempt", "list_mock_attempts", "save_mock_evaluation", "save_ideal_solution", "save_simple_solution", "get_lld_feedback", "log_lld_drill", "get_lld_drill_log", "get_current_time", + "start_hld_mock_attempt", "save_hld_attempt", "save_hld_diff", + "save_hld_evaluation", "list_hld_mock_attempts", "read_hld_mock_file", + "get_hld_feedback", }