release 0.0.100: large-source-package platform fixes (#247 #248 #249) + build.mcpp capability completion#250
Merged
Merged
Conversation
…on Windows Driver-style toolchains (g++/clang++ as link driver — the gnu dialect, which is what Windows clang-MSVC resolves to via dialect_for) inlined $in into cxx_link/cxx_archive/cxx_shared. On Windows ninja spawns via CreateProcess (32 KiB command-line ceiling), so ffmpeg/opencv-class packages (thousands of objects) overflow at the final link. Route $in through @$out.rsp on Windows only (gcc/clang drivers and GNU/llvm ar all accept @rspfile); POSIX rule text stays byte-identical (ARG_MAX is ample). The three rules share one driver_rule emitter. Also lands the batch design doc (P0-P2).
…ction The build path splices only the running host's per-OS block and skip- tables the rest, so a typo in the windows section is invisible on linux CI. --all-os re-parses the descriptor once per OS the xpm table declares versions for (a linux+macosx-only package legitimately has no windows section), via a new osOverride seam on synthesize_from_xpkg_lua — same function, same grammar as the resolver. mcpp-index CI can now lint all legs of a multi-platform descriptor from one runner (design doc 2026-07-19 §4).
…path (launcher-unify) Root cause: on macOS the shell fallback built `KEY='v' cd <cwd> && prog 2>&1` when both extraEnv and cwd were non-empty. POSIX binds the env assignment prefix to the first simple command only — `cd` — so across `&&` the real program received NO extra env. The only env+cwd call site is the build.mcpp child (build_program.cppm), which therefore lost the entire MCPP_* G3 contract (MCPP_OUT_DIR etc.) on macOS and aborted. Fix: extend the run_exec / capture_exec guards from __linux__ to __linux__ || __APPLE__ so macOS shares the direct posix_spawn path — child-only env via merged_environ, cwd via posix_spawn_file_actions_addchdir_np (macOS 10.15+; mcpp's floor is macOS 14, no runtime probe needed). Portability: on Apple, `environ` is only linkable from executables, not dylibs, so merged_environ now reads the host block through a host_environ() helper using _NSGetEnviron() from <crt_externs.h>; Linux keeps the plain symbol. Windows _WIN32 branches and the residual non-POSIX shell fallback are untouched; the L286 rationale + TODO(launcher-unify) now document Windows as the sole remaining shell-path exception. As a side effect macOS also sheds the shell quoting/signal/injection surface for these two launchers. Regression test: capture_exec with BOTH non-empty extraEnv and cwd (previously the untested combination — all prior call sites passed at most one) asserting the child sees the env value AND runs in cwd; runs on Linux (same shared code path) and macOS CI.
…keeps incremental builds alive) materialize_generated_files rewrote every entry unconditionally whenever prepare_build ran -> mtime bump -> ninja (mtime-driven, header reached via depfiles) recompiled every TU including the materialized header, even when only an unrelated source changed. Frozen-snapshot packages (compat.ffmpeg: config.h included by 2283 TUs) degraded to full rebuilds on every edit. Now the intended content is byte-compared against the on-disk file and the write is skipped when identical, preserving the mtime. Correctness is unchanged: content changes are already folded into the fingerprint, which owns change detection; the skip only avoids the mtime churn (mirroring the build.mcpp cache design). Path-escape guards, create_directories, and the fingerprint interplay are untouched. e2e 142 locks it: touching an unrelated TU (abandoning the P0 fast-path so prepare re-runs) must keep the generated header's mtime and not recompile the including TU; changing the generated content in mcpp.toml must still rewrite the file and rebuild (no false-fresh). A truly no-op rebuild never reaches materialization (fast-path), so the unrelated-touch case is the load-bearing repro - verified failing on the unfixed binary.
…stop shadowing system headers) Root cause: dependency include_dirs are always emitted as -I (highest priority). On case-insensitive macOS filesystems a dep's extracted-tarball root on -I that contains a file named like a standard header shadows it for every C++ TU of every consumer — ffmpeg's top-level VERSION file shadows libc++'s <version>, breaking any consumer that touches the standard library. There was no lower-priority include variant in the descriptor vocabulary. Why -idirafter and not -isystem: -isystem dirs still precede the default system dirs, so the poison file still wins. -idirafter dirs are searched AFTER all system dirs (gcc+clang), so system headers win while non-system headers (<libavutil/frame.h>) are still found. Full chain: - manifest/types.cppm: BuildConfig.includeDirsAfter next to includeDirs - manifest/xpkg.cppm: include_dirs_after key (+ include_after / include_dir_after aliases), parse branch mirroring include_dirs; platform overlays get it for free via the shared key loop - manifest/toml.cppm: [build] include_dirs_after - modgraph: UsageRequirements.includeDirsAfter, SourceUnit.localIncludeDirsAfter, threaded through scan_package / scan_packages / scan_packages_p1689 with the same expand_dir_glob treatment (`*` extracted-tarball-root convention) - build/prepare.cppm: expansion into privateBuild/publicUsage, propagation along the SAME dependency edges as includeDirs — consumers receive them as after-dirs, never upgraded to -I; after-dirs enter the resolved-usage identity string; staged-secondary absolutization mirrored - build/plan.cppm: CompileUnit.localIncludeDirsAfter populated wherever localIncludeDirs is (scanned units + synthesized entry main) - build/ninja_backend.cppm: emitted as -idirafter into $local_includes, appended after the -I entries (flag semantics, not position, carry the priority). MSVC dialect: cl.exe has no -idirafter — documented degradation to trailing /I (clang-MSVC uses the gnu dialect and gets real -idirafter) - build/flags.cppm: main-manifest [build] include_dirs_after on the global typed-path channel (-idirafter / trailing /I; nasm degrades to -I) - build/compile_commands.cppm: -idirafter spelling preserved in the compile DB so clangd reproduces the search order Adjacent consistency fix: plan.cppm local_include_dirs_for_manifest did a bare root/inc join with no glob expansion, unlike the dep path in prepare.cppm. Now unified on expand_dir_glob, with a plain-join fallback for literal wildcard-free entries that don't exist yet (expand_dir_glob only returns existing dirs; the old unconditional join must keep working for dirs created later by build steps). Tests: unit (ninja -idirafter emission + ordering, msvc /I degradation, toml + xpkg parse), e2e 141 (poisoned stdlib.h + real mylib.h in a path dep's include root: include_dirs_after builds and runs, plain include_dirs control fails on the poison).
Names the capability that the grey `mcpp:generated=<absolute path>` usage
provided by accident: a build.mcpp can now SELECT a pre-existing source
(tarball payload / vendored tree) into the compile set, without claiming it
wrote the file. Design: 2026-07-19-large-source-pkg-platform-fixes-and-
buildmcpp-generation-design.md §3.1 item 1.
- parse_line: new `source` directive key; value verbatim into Directives::sources.
- Path rules: absolute taken as-is; relative resolves against the PACKAGE
ROOT in both root and dependency mode (unlike generated=, whose dep-mode
relatives resolve to genBase/OUT_DIR) — a payload file lives in the package
tree, never in OUT_DIR. `generated=` behavior is byte-for-byte untouched,
including the absolute-path grey usage.
- apply(): same landing as generated= (bc.sources + m.modules.sources, both
lists so the scanner sees files outside the base globs).
- Existence: post-run hard error with a source=-specific message ("selected
source ... no such file"); cache_fresh also invalidates when a selected
source vanishes (mirrors generated=).
- Cache: new `d source <path>` record kind (write_cache/read_cache round-trip).
- Typed lib: mcpp::source(path) mirrors mcpp::generated 1:1.
- execute.cppm sources_newer_than: the root build.mcpp is now a fast-path
input — editing ONLY build.mcpp previously never fell through to
prepare_build (fresh build.ninja short-circuited before the program-hash
cache key could see the change), so the documented "re-runs when the
build.mcpp source itself changes" was unreachable on the fast path. Found
by the new e2e's missing-source case; covers try_fast_build AND try_fast_run.
- Docs: directive + typed-API tables (en/zh).
No unit seam exists for parse_line/apply (anonymous namespace in the module
impl unit) — covered by e2e 143_build_mcpp_source_directive.sh: pre-existing
file outside all sources globs is compiled+linked via the typed API, `d source`
cache record replays on rebuild without a re-run, and a missing selection
fails with the actionable error.
Verified: unit 35/35; e2e 143/110/111/125 green.
Two new directives (design §3.1 item 2) replacing the unnormalized `cxxflag=-I` + `cflag=-I` double-emission hack (opencv descriptor:412-413): - Paths resolve like link-search: absolute as-is, relative against the package root, lexically normalized at parse time (abs_against_root). - include-dir → buildConfig.includeDirs; include-dir-after → buildConfig.includeDirsAfter. The includeDirsAfter FIELD is added to manifest/types.cppm here (line coordinated with the parallel #249 worktree); the -idirafter NINJA EMISSION chain is deliberately NOT implemented — that worktree owns ninja_backend/plan. This commit lands values into buildConfig only; the root-path end-to-end -idirafter arrives when the branches merge. - Cache: `d include-dir` / `d include-dir-after` record kinds round-trip. - Typed lib: mcpp::include_dir / mcpp::include_dir_after. Scope decision (dep path — the invariant is: directive include dirs NEVER leak to consumers): a dep's own TUs read privateBuild, consumers read publicUsage (re-flowed by the usage fixpoint right after the dep loop). The dep loop therefore mirrors the new includeDirs tail into pkg.privateBuild.includeDirs ONLY (publicUsage untouched → no leak; the post-snapshot bcDep entries are inert since descriptor include_dirs propagation snapshotted publicUsage at makePackageRoot). includeDirsAfter has no privateBuild slot (emission chain is #249's), so for deps it is spelled as private "-idirafter <dir>" flag pairs into privateBuild cflags+cxxflags — deps get the semantics today, same two-token spelling as host_base_flags. For the ROOT the dirs join buildConfig before the packages[0] snapshot, where expandIncludeDirs handles absolute entries; root has no consumers, so privacy holds trivially. Also folded here (nearest theme — unsupported per-feature include_dirs was the motivating case): xpkg feature UNKNOWN SUB-KEY is now recorded as `features.<name>.<sub>` in xpkgUnknownKeys instead of being silently swallowed; the existing adoption-site warn_unknown_xpkg_keys (prepare.cppm) surfaces it as a warning. Unit: XpkgUnknownKeys.FeatureSubKeyRecorded. e2e 144_build_mcpp_include_dir.sh: header written under MCPP_OUT_DIR + absolute include-dir → #include resolves with no manual -I; include-dir-after accepted (no unknown-directive warning); both replay from the directive cache. Verified: unit 35/35 green; e2e 144/143/110/111/125 green.
Cargo CARGO_CFG_TARGET_* parity (design §3.1 item 3): contract_env now parses the resolved target ONCE through the canonical triple parser (mcpp.toolchain.triple — already imported for MCPP_HOST) and injects MCPP_TARGET_OS linux|macos|windows MCPP_TARGET_ARCH GNU spelling (x86_64, aarch64, ...) MCPP_TARGET_ENV gnu|musl|msvc, "" when the triple has no env segment so every build.mcpp (ffmpeg/opencv per-OS selection) stops hand-splitting MCPP_TARGET. All three are always SET; an escape-hatch triple outside the canonical vocabulary yields "" for all three (parse failure is not an error here — MCPP_TARGET still carries the verbatim spelling). Re-run key: verified — contract_hash hashes the WHOLE (name,value) env vector, and the new vars ride that same vector, so they fold into ctxHash with no extra plumbing (a target change already re-ran the program via MCPP_TARGET; the splits add no new invalidation axis). Typed lib: mcpp::target_os()/target_arch()/target_env() readers. e2e 110 extended: asserts the three vars are set, OS/ARCH agree with the full MCPP_TARGET triple's leading segments, and ENV is set (content may be empty on macOS). Docs env tables updated (en/zh). Verified: unit 35/35; e2e 110/143/144/111/125 green.
…CPP_DEP_*_DIR Design §3.1 item 4 (+ #230-#243 ledger follow-up): the ROOT project's build.mcpp used to run BEFORE dependency resolution (prepare.cppm ~1180), so unlike a dependency's it never saw MCPP_DEP_<NAME>_DIR — consumer-side synthesis (ffmpeg-m reading compat.ffmpeg's payload tree) was impossible. The call now sits right after the dep build.mcpp loop / usage fixpoint / capability binding, BEFORE the [targets.*] gate and the modgraph scan, and populates env.depDirs from the SAME authoritative dependencyEdges graph as the dep path (consumer index 0 = root; canonical + namespace-stripped spellings; same collision guard in contract_env). Ordering invariants, each verified at the new call point: (1) generated=/source= registration precedes the modgraph scan — the scan walks packages[0].manifest, so the new-source TAILS are mirrored into packages[0].manifest.{buildConfig.sources,modules.sources} (see wrinkle below); the scan call sits ~200 lines further down. (2) directive flags precede canonicalization/fingerprint — fpi.compileFlags = canonical_compile_flags(*m) + canonical_package_build_metadata(packages) is computed AFTER the scan, i.e. after this call; flag tails are mirrored into packages[0].privateBuild (per-TU assembly) AND packages[0].manifest.buildConfig (fingerprint metadata parity). (3) materialize_generated_files (root) still runs earlier — untouched at its pre-move position, so [generated_files] may still produce build.mcpp. (4) L1 cfg-conditional merge ordering unchanged — it still runs before makePackageRoot; ONLY the build.mcpp call moved later (e2e 108 green). (5) root features env — recomputed with the byte-identical expression (feature_closure(*m, parse_feature_request(overrides.features))) so the contract hash / build.mcpp cache is stable across the move. The wrinkle the old ordering hid: apply() used to mutate *m before `packages[0] = makePackageRoot(*root, *m)` snapshotted buildConfig into privateBuild/manifest — the copies the scan and per-TU flag assembly read. Post-move the snapshot (and root feature activation on it) already happened, so the directive tails are mirrored explicitly, dep-loop style: sources → manifest.buildConfig.sources + manifest.modules.sources; c/cxx flags → privateBuild + manifest.buildConfig; ldflags → linkUsage + manifest.buildConfig (final link reads *m, already applied); include-dir → privateBuild.includeDirs ONLY (private discipline — under the old order they also leaked into root publicUsage via the snapshot, harmless with no consumers but now deliberately private); include-dir-after → manifest.buildConfig.includeDirsAfter. Known benign delta: directive flags now append AFTER feature/usage-propagated flags in privateBuild instead of before them (order within one flags vector). e2e 145_root_build_mcpp_dep_dirs.sh (modeled on 125): root with a path dep; the ROOT's build.mcpp hard-fails unless dep_dir("datad") is set, generates a source returning it; asserts the dir exists and is datad's root. Docs: MCPP_DEP_<NAME>_DIR documented in the env table (en/zh) — previously undocumented — with the root now receiving it. Verified: unit 35/35; e2e 145/110/111/112/125/143/144 green + regression batch 106/107/108/109/114/100/04/09/15/126/128 green.
…249 channel Review pass over the merged P0+P1 branch. Two spellings of the same decision survived the parallel tracks: - dep loop: after-dirs were spelled as raw private '-idirafter <dir>' flag pairs (written before #249 landed privateBuild.includeDirsAfter); the raw spelling bypasses the typed channel's per-dialect degradations (cl.exe /I, NASM -I). Now appendUniquePath into the typed slot. - root: after-dirs reached only the fingerprint mirror (manifest.buildConfig), never privateBuild — scanned units read privateBuild.includeDirsAfter, so 'mcpp:include-dir-after=' on the root was inert on compile edges. Both channels now mirrored for both kinds, matching the flag mirrors above. e2e 144 upgraded from 'directive accepted' to asserting -idirafter reaches build.ninja.
… directive-fold owner, type/rule unification Consolidated fixes from the three-lens pre-PR review (correctness / cross-platform / architecture): - #247 P0: escape_ninja_path emits generic (forward-slash) node names. rspfile_content copies node names verbatim into a response file that gcc/clang/GNU ar tokenize GNU-style, where backslash is an ESCAPE — obj\cli.o would arrive as objcli.o and break every Windows driver-style link. All Windows consumers accept forward slashes; POSIX output is byte-identical. - link/archive/shared rules: ONE link_rule emitter for both dialect branches (useRsp = separateLinker || is_windows); msvc rule text byte-identical to the previous hand-written stanzas. - prepare.cppm: markDirectiveTail/foldDirectiveTailIntoPrivateBuild — a single owner of 'which compile-visible channels a build.mcpp directive lands in', shared by the dep loop and the root call site (kills the #242 two-derivations shape the parallel tracks had reintroduced). Link/source/fingerprint residues stay at the call sites where they genuinely differ. - BuildConfig::includeDirsAfter aligned to filesystem::path (was string), deleting per-consumer conversions. - local_include_flags derives the NASM degradation from cu.source itself instead of a third parameter threaded by every caller. - Windows shell fallback: cd /d (cmd.exe cd does not switch drives). - docs: source= joins the cache-invalidation list; 0.0.100+ tags on the new directives/env rows; NASM -I degradation documented (en+zh).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
大型源码直编包(ffmpeg/opencv 级,数千 TU)全平台化批次。设计文档:
.agents/docs/2026-07-19-large-source-pkg-platform-fixes-and-buildmcpp-generation-design.md(三 issue 根因逐行核实 + 第四坑 + P0–P2 方案)。Closes #247, closes #248, closes #249.
P0 平台修复
link_rule发射器;ninja 节点名generic_string()(rsp 按 GNU 文法分词,反斜杠是转义符)。POSIX 规则文本逐字节不变。capture_exec/run_execlauncher-unify 到 posix_spawn(child-only env +addchdir_np,_NSGetEnviron()portable-correct)。根因:shell 前缀 env 只绑给cd,全仓唯一 env+cwd 双非空调用点即 build.mcpp。附 windowscd /d。include_dirs_after→-idirafter全链(双文法、glob、传播不升级);方言降级单点:cl.exe→/I、NASM→-I。附主工程 include glob 展开与 dep 路径统一。generated_files无条件重写 → mtime 抖动 → 冻结快照包每次全量重编;现内容相同跳写。P1 build.mcpp 指令面补全
mcpp:source=/mcpp:include-dir[-after]=(私有,typed 通道)/MCPP_TARGET_OS/ARCH/ENV/ root build.mcpp 后移依赖解析后并获MCPP_DEP_*_DIR;指令落通道收敛 root/dep 共享单 fold;feature 未知子键告警;xpkg parse --all-os。架构 review(三视角,PR 前完成)
正确性(零高置信 bug)、跨平台(抓到 rsp 反斜杠 P0 与
cd /d)、架构/简洁性(fold 单一 owner、类型对齐 path、规则发射统一、文档版本标注)。测试
发布
末 commit 为
chore(release): 0.0.99 -> 0.0.100(CHANGELOG + 双版本常量)。合入后走 tag release → 镜像 → xim-pkgindex → 真装验证 → pin bump。