SourceFile text does not include BOM

[acutmore]

Steps to reproduce

The sourceFile.{text,getFullText(),getText()} properties return a string that does not include the BOM from the original file, however the node positions are relative to the string with the BOM. This means the node positions and their text do not align.

import * as ts from "@typescript/native-preview/unstable/ast";
import { createVirtualFileSystem } from "@typescript/native-preview/unstable/fs";
import { API } from "@typescript/native-preview/unstable/sync";

const UTF8_BOM = "\uFEFF";
const INPUT = `${UTF8_BOM}let answer:number=42;`;

const fs = createVirtualFileSystem({
    "/tsconfig.json": JSON.stringify({ files: ["/input.ts"] }),
    "/input.ts": INPUT,
});
const api = new API({
    cwd: "/",
    fs,
});
const snap = api.updateSnapshot({ openProject: "/tsconfig.json" });
const ast = snap.getProject("/tsconfig.json")!.program.getSourceFile("/input.ts")!;

walk(ast);

function walk(node: ts.Node) {
    const start = node.getFullStart();
    const end = node.getEnd();
    const src = node.getSourceFile().getText();
    console.log({ start, end, text: src.slice(start, end), text2: node.getFullText() });
    node.forEachChild(walk);
}

Behavior with typescript@6.0

import * as ts from "typescript";

const UTF8_BOM = "\uFEFF";
const INPUT = `${UTF8_BOM}let answer:number=42;`;

const ast = ts.createSourceFile("input.ts", INPUT, ts.ScriptTarget.ESNext, /* setParentNodes */true);

walk(ast);

function walk(node: ts.Node) {
    const start = node.getFullStart();
    const end = node.getEnd();
    const fullText = node.getSourceFile().getFullText();
    console.log({ start, end, text: fullText.slice(start, end), text2: node.getFullText() });
    node.forEachChild(walk);
}

Prints:

...
{ start: 4, end: 11, text: ' answer', text2: ' answer' }
{ start: 12, end: 18, text: 'number', text2: 'number' }
{ start: 19, end: 21, text: '42', text2: '42' }
{ start: 22, end: 22, text: '', text2: '' }

Behavior with tsgo

...
{ start: 4, end: 11, text: 'answer:', text2: 'answer:' }
{ start: 12, end: 18, text: 'umber=', text2: 'umber=' }
{ start: 19, end: 21, text: '2;', text2: '2;' }
{ start: 22, end: 22, text: '', text2: '' }

[jakebailey]

I'm pretty sure that if you had read the same file using 6.0's ts.sys, we'd strip the BOM and you'd get the Corsa like behavior. Not sure we considered that anyone would actually put a BOM into the raw text...

[jakebailey]

In a way this implies we should also be BOM stripping when creating source files too, though reasonably the only time this would happen is for the UTF-8 BOM when manually constructed like this?

[acutmore]

I actually noticed this when running the ts-blank-space tests which was loading test fixtures off disk and then running them through ts-blank-space (string -> string) type stripping. The test was just passing the result of fs.readFileSync into the transform.

also be BOM stripping when creating source files too

This is what I was imagining, so that it's consitant that the node positions align with the node text.

I can work around this easily enough by stripping the BOM. Just wanted to point it out as a technical difference from how Strada behaved in case others hit the same subtle difference.

[jakebailey]

I assume that if you read instead via ts.sys.readFile, the BOM is gone? (Not that I am advocating for creating that dependence for you.)

I do think we need to probably drop the BOM (pun intended) somewhere else

[helenkwok]

I reproduced this independently while building a downstream parser-compatibility harness against the current native preview.

With input \uFEFFexport const bomValue = 1;, the project-backed sync API currently reports:

Observation	Result
input begins with BOM	true
SourceFile.text begins with BOM	false
SourceFile.getFullText() begins with BOM	false
first statement range	0..27
text.slice(start, end)	"export const bomValue = 1;\n"
expected slice	"export const bomValue = 1;"

This matters for parser/linter consumers because AST offsets and returned source text cannot be used together safely.

The reproduction is exercised nightly in https://github.com/helenkwok/typescript-parser-compat via fixtures/bom.ts, scripts/native-project-probe.mjs, and test/native-project-adapter.test.mjs. The harness keeps this as a sentinel so an upstream behavior change will be detected automatically.