Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions src/main/java/io/jawk/Cli.java
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ public final class Cli {
private File compileOutputFile;
private boolean printUsage;
private boolean sandbox;
private boolean processStandardInput;
private boolean disableOptimize;
private boolean profiling;
private File profilingOutputFile;
Expand Down Expand Up @@ -544,6 +545,7 @@ private void executeProgram(Awk awk, AwkProgram program, File memoryFile) throws
avm.setAwkSink(sink);
avm.setErrorStream(err);
avm.setWarningStream(err);
avm.getJrt().setSpawnedProcessesInheritStandardInput(processStandardInput);
if (memoryFile != null) {
restorePersistentMemoryIfPresent(avm, memoryFile);
}
Expand Down Expand Up @@ -754,12 +756,30 @@ public static Cli create(String[] args, InputStream is, PrintStream os, PrintStr

/**
* Entry point for the command-line interface.
* <p>
* This method carries process semantics: because it is what the
* {@code jawk} launch of the JVM runs, it declares that the standard input
* the CLI reads is the standard input of the process, and the children of
* {@code system()} and of command pipes may inherit it, as POSIX requires.
* No Java-side check can distinguish the launcher-installed
* {@code System.in} from a replacement installed with
* {@code System.setIn}, so that declaration is part of this method's
* contract: code that replaces {@code System.in} must not call
* {@code main} programmatically — embedders and tests go through
* {@link #create(String[], InputStream, PrintStream, PrintStream)}, the
* constructors, or the {@link Awk} API, where spawned children always get
* a closed standard input instead.
*
* @param args command-line arguments
*/
public static void main(String[] args) {
try {
Cli cli = new Cli();
// Only the process entry point can vouch that the standard input
// this CLI reads is the standard input of the JVM itself, the one
// stream a spawned child can inherit; embedders and tests reach
// run() through other paths and never set this.
cli.processStandardInput = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Do not assume every main invocation uses native stdin

When an embedder or test calls the public Cli.main after System.setIn(customStream), this unconditional flag makes system() and command-pipe children inherit the process's native descriptor 0 even though Jawk itself reads the replacement stream, so a child can block on or consume unrelated host input. The repository already invokes Cli.main programmatically in AwkTest.java, so main is not exclusively an OS launch path. The fresh evidence relative to the earlier finding is this new unconditional assignment; eligibility must only be enabled when the CLI input is actually backed by native stdin.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that no capture or flag can survive a programmatic Cli.main call after System.setIn — and there is no supported Java-side check that can tell the launcher-installed System.in from a replacement (the fd0 backing of a stream is not observable without deep reflection into java.io). So the boundary is now drawn as a documented contract instead: Cli.main carries process semantics — it is what the jawk launch of the JVM runs, and its children may inherit the process's standard input, as POSIX requires of awk. Code that replaces System.in must go through create(), the constructors, or the Awk API, where spawned children always get a closed standard input. The one programmatic Cli.main call in the test suite (AwkTest.compileTuplesViaCLI) now goes through create() accordingly, so nothing in the repository calls main programmatically anymore.

cli.parse(args);
cli.run();
} catch (ExitException e) {
Expand Down
88 changes: 66 additions & 22 deletions src/main/java/io/jawk/jrt/JRT.java
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,8 @@ public class JRT {
* same data as the main input loop does when no operand is given.
*/
private InputStream standardInput = System.in;

private boolean spawnedProcessesInheritStandardInput;
/**
* Sink writing to the standard error of the process, used by the
* {@code /dev/stderr} special filename; created on first use and discarded
Expand Down Expand Up @@ -3167,20 +3169,51 @@ public boolean jrtConsumeFileInput(String fileNameParam) throws IOException {
return true;
}

private static Process spawnProcess(String cmd) throws IOException {
Process p;

if (IS_WINDOWS) {
// spawn the process using the Windows shell
ProcessBuilder pb = new ProcessBuilder("cmd.exe", "/c", cmd);
p = pb.start();
} else {
// spawn the process using the default POSIX shell
ProcessBuilder pb = new ProcessBuilder("/bin/sh", "-c", cmd);
p = pb.start();
private static Process spawnProcess(String cmd, boolean inheritStandardInput) throws IOException {
ProcessBuilder pb = IS_WINDOWS
// spawn the process using the Windows shell
? new ProcessBuilder("cmd.exe", "/c", cmd)
// spawn the process using the default POSIX shell
: new ProcessBuilder("/bin/sh", "-c", cmd);
if (inheritStandardInput) {
pb.redirectInput(ProcessBuilder.Redirect.INHERIT);
}
return pb.start();
}

/**
* Declares whether processes spawned on behalf of the script share the
* standard input of this JVM. POSIX gives the children of {@code system()}
* and of a command pipe the same standard input as awk itself, which is how
* terminal-aware commands like {@code "stty size" | getline} find the
* controlling terminal. That is only faithful when Jawk reads the real
* standard input of the process, which no capture of {@code System.in} can
* establish — an embedder may have replaced the stream with
* {@code System.setIn} at any point, including before this class
* initializes — so eligibility is asserted explicitly by the one caller
* that can vouch for it: the command-line entry point of the process.
* Everywhere else the flag stays {@code false} and the child's standard
* input is closed, since a Java stream cannot be lent to another OS
* process, and exposing the host JVM's real descriptor 0 instead would
* leak input the embedder never gave to Jawk.
*
* @param inherit {@code true} when the standard input this run reads is
* the standard input of the JVM process itself
*/
public void setSpawnedProcessesInheritStandardInput(boolean inherit) {
this.spawnedProcessesInheritStandardInput = inherit;
}

return p;
/**
* Tells whether processes spawned on behalf of the script share the
* standard input of this JVM, as declared through
* {@link #setSpawnedProcessesInheritStandardInput(boolean)}.
*
* @return {@code true} when spawned processes inherit the JVM's standard
* input
*/
private boolean spawnedProcessInheritsStandardInput() {
return spawnedProcessesInheritStandardInput;
}

/**
Expand Down Expand Up @@ -3261,8 +3294,13 @@ private CommandInputState createCommandInputState(String cmd) throws IOException
Process process = null;
Thread errorPump = null;
try {
process = spawnProcess(cmd);
process.getOutputStream().close();
// POSIX: the child shares awk's standard input; when that is not
// possible (embedded execution on a custom stream) it stays closed
boolean inheritStandardInput = spawnedProcessInheritsStandardInput();
process = spawnProcess(cmd, inheritStandardInput);
if (!inheritStandardInput) {
process.getOutputStream().close();
}
errorPump = DataPump.dumpAndReturnThread(cmd + " stderr", process.getErrorStream(), error);
PartitioningReader reader = new PartitioningReader(
new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8),
Expand Down Expand Up @@ -3294,7 +3332,8 @@ private ProcessOutputState createProcessOutputState(String cmd) {
PrintStream processOutput = null;
try {
processOutput = awkSink.getPrintStream();
process = spawnProcess(cmd);
// the pipe itself is the child's standard input
process = spawnProcess(cmd, false);
stderrPump = DataPump.dumpAndReturnThread(cmd + " stderr", process.getErrorStream(), error);
stdoutPump = DataPump.dumpAndReturnThread(cmd + " stdout", process.getInputStream(), processOutput);
PrintStream processInput = new PrintStream(process.getOutputStream(), true, StandardCharsets.UTF_8.name());
Expand Down Expand Up @@ -3502,10 +3541,11 @@ private boolean jrtCloseCommandReader(String cmd) {
* Executes the command specified by cmd and waits
* for termination, returning an Integer object
* containing the return code.
* stdin to this process is closed while
* threads are created to shuttle stdout and
* stderr of the command to stdout/stderr
* of the calling process.
* The command inherits the standard input of the JVM when Jawk reads the
* real standard input (CLI runs), as POSIX requires of {@code system()};
* otherwise its standard input is closed. Threads are created to shuttle
* stdout and stderr of the command to stdout/stderr of the calling
* process.
*
* @param cmd The command to execute.
* @return Integer(return_code) of the created
Expand All @@ -3514,9 +3554,13 @@ private boolean jrtCloseCommandReader(String cmd) {
public Integer jrtSystem(String cmd) {
try {
PrintStream processOutput = awkSink.getPrintStream();
Process p = spawnProcess(cmd);
// no input to this process!
p.getOutputStream().close();
// POSIX: the child shares awk's standard input; when that is not
// possible (embedded execution on a custom stream) it stays closed
boolean inheritStandardInput = spawnedProcessInheritsStandardInput();
Process p = spawnProcess(cmd, inheritStandardInput);
if (!inheritStandardInput) {
p.getOutputStream().close();
}
Thread errorPump = DataPump.dumpAndReturnThread(cmd + " stderr", p.getErrorStream(), error);
Thread outputPump = DataPump.dumpAndReturnThread(cmd + " stdout", p.getInputStream(), processOutput);
boolean interrupted = false;
Expand Down
10 changes: 10 additions & 0 deletions src/site/markdown/behavior-changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,16 @@ released version automatically via .github/scripts/stamp-behavior-changes.sh.
`-f prog.awk`, `-v x=1`, `-F :`, as in gawk, mawk, BWK awk, and goawk. Previously the glued
form was rejected with `Unknown parameter`
([#574](https://github.com/jawkio/jawk/issues/574)).
- The children of `system(cmd)` and of a command input pipe (`"cmd" | getline`) now inherit
Jawk's standard input when Jawk reads the standard input of the JVM (every CLI run), as POSIX
requires and as gawk, mawk, and BWK awk behave: `echo hi | jawk 'BEGIN { "sort" | getline l;
print l }'` prints `hi`, and terminal-aware commands such as `"stty size" | getline` can reach
the controlling terminal. Previously the child's standard input was always closed, so stdin
filters read nothing and `stty` failed with `Inappropriate ioctl for device`. Output pipes
(`print | "cmd"`) are unchanged — the pipe itself remains the child's standard input — and
embedded executions bound to a custom Java input stream keep the closed-stdin behavior, since
a Java stream cannot be lent to another OS process
([#575](https://github.com/jawkio/jawk/issues/575)).
- An input-derived value whose text is a number surrounded by blanks — a record like `" 12 "`,
a `getline var` result read from padded input, a `split()` piece under a non-default
separator — is now recognized as a POSIX numeric string, so `$0 == 12` is true for the record
Expand Down
9 changes: 9 additions & 0 deletions src/site/markdown/java-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,15 @@ awk.script("BEGIN { system(\"mycommand\") }")
The CLI uses `.errorStream(System.err)` so that command errors appear on the
console rather than mixing with normal output.

Subprocess **stdin** follows POSIX in CLI runs: the children of `system("...")` and of a
command input pipe (`"cmd" | getline`) inherit the standard input of the JVM, so stdin
filters and terminal-aware commands (`"stty size" | getline`) work as they do under gawk.
In embedded runs — any execution not started through the `jawk` command line — the
child's standard input is closed instead: a Java stream cannot be lent to another OS
process, and the host's real standard input is never handed to the script's children.
The child of an output pipe (`print ... | "cmd"`) always reads the pipe itself as its
standard input.

## See Also

- [Java Quickstart](java.html)
Expand Down
10 changes: 9 additions & 1 deletion src/test/java/io/jawk/AwkTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -1594,7 +1594,15 @@ public void loadSerializedProgram() throws Exception {
@Test
public void compileTuplesViaCLI() throws Exception {
File tmp = File.createTempFile("jawk", ".tpl");
Cli.main(new String[] { "-K", tmp.getAbsolutePath(), "{ print toupper($0) }" });
// Cli.main is reserved for the process launch (its children may inherit
// the JVM's standard input); programmatic callers go through create()
Cli
.create(
new String[]
{ "-K", tmp.getAbsolutePath(), "{ print toupper($0) }" },
new ByteArrayInputStream(new byte[0]),
System.out,
System.err);

Cli cli = Cli.parseCommandLineArguments(new String[] { "-L", tmp.getAbsolutePath() });

Expand Down
58 changes: 58 additions & 0 deletions src/test/java/io/jawk/AwkTestSupport.java
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import java.util.stream.Collectors;
import java.util.stream.Stream;
Expand Down Expand Up @@ -138,6 +139,63 @@ public static Path sharedTempDirectory() {
return SHARED_TEMP_DIR;
}

/**
* Runs a script through the CLI in a freshly spawned JVM whose standard
* input is redirected from a file holding the given content, asserts that
* the run succeeds, and returns its standard output with platform line
* separators normalized to {@code \n}.
* <p>
* The in-process builders cannot observe behavior that only exists across a
* real process boundary, such as the file-descriptor inheritance of spawned
* children; this helper exists for exactly those tests. The script travels
* through a file because Windows mangles embedded double quotes in inline
* command-line arguments, and the child's standard output and error are
* redirected to files so no pipe can fill up and stall either side: the
* timeout is enforced by {@code waitFor} alone, and an expired child is
* killed.
*
* @param description human readable description used in assertion messages
* @param script the AWK program to run
* @param stdinContent the bytes offered to the JVM as its standard input
* @return the standard output of the run, with {@code \r\n} normalized to
* {@code \n}
* @throws Exception when the JVM cannot be spawned or its output read
*/
public static String runCliInFreshJvm(String description, String script, String stdinContent) throws Exception {
Path directory = Files.createTempDirectory(SHARED_TEMP_DIR, "cli-jvm");
Path stdinFile = directory.resolve("stdin.txt");
Files.write(stdinFile, stdinContent.getBytes(StandardCharsets.UTF_8));
Path scriptFile = directory.resolve("script.awk");
Files.write(scriptFile, script.getBytes(StandardCharsets.UTF_8));
Path stdoutFile = directory.resolve("stdout.txt");
Path stderrFile = directory.resolve("stderr.txt");

String javaBinary = new File(new File(System.getProperty("java.home"), "bin"), "java").getAbsolutePath();
File classes = new File(Cli.class.getProtectionDomain().getCodeSource().getLocation().toURI());
ProcessBuilder pb = new ProcessBuilder(
javaBinary,
"-cp",
classes.getAbsolutePath(),
Cli.class.getName(),
"-f",
scriptFile.toAbsolutePath().toString());
pb.redirectInput(stdinFile.toFile());
pb.redirectOutput(stdoutFile.toFile());
pb.redirectError(stderrFile.toFile());

Process process = pb.start();
boolean finished = process.waitFor(30, TimeUnit.SECONDS);
if (!finished) {
process.destroyForcibly();
process.waitFor();
}
String stderr = new String(Files.readAllBytes(stderrFile), StandardCharsets.UTF_8);
assertEquals(description + ": CLI JVM timed out; stderr: " + stderr, Boolean.TRUE, Boolean.valueOf(finished));
assertEquals(description + ": CLI JVM failed; stderr: " + stderr, 0, process.exitValue());
String stdout = new String(Files.readAllBytes(stdoutFile), StandardCharsets.UTF_8);
return stdout.replace("\r\n", "\n");
}

/**
* Represents a fully configured test case produced by one of the builders.
* Implementations know how to prepare the execution environment, run the
Expand Down
74 changes: 74 additions & 0 deletions src/test/java/io/jawk/SpawnedProcessStdinTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
package io.jawk;

/*-
* ╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲
* Jawk
* ჻჻჻჻჻჻
* Copyright (C) 2006 - 2026 MetricsHub
* ჻჻჻჻჻჻
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Lesser Public License for more details.
*
* You should have received a copy of the GNU General Lesser Public
* License along with this program. If not, see
* <http://www.gnu.org/licenses/lgpl-3.0.html>.
* ╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱╲╱
*/

import static org.junit.Assert.assertEquals;

import org.junit.Test;

/**
* Verifies that the children of {@code system()} and of a command input pipe
* ({@code "cmd" | getline}) inherit Jawk's standard input, as POSIX requires,
* when Jawk reads the standard input of the JVM — and that embedded executions
* bound to a custom stream keep the child's standard input closed.
* <p>
* The inheritance can only be observed across a real process boundary, so the
* inheriting cases run through
* {@link AwkTestSupport#runCliInFreshJvm(String, String, String)}, which
* spawns the CLI in a fresh JVM whose standard input is redirected from a
* file.
*/
public class SpawnedProcessStdinTest {

@Test
public void commandInputPipeChildReadsJawkStandardInput() throws Exception {
String output = AwkTestSupport
.runCliInFreshJvm(
"cmd|getline child reads Jawk stdin",
"BEGIN { \"sort\" | getline line; print \"[\" line \"]\" }",
"hello\n");
assertEquals("[hello]\n", output);
}

@Test
public void systemChildReadsJawkStandardInput() throws Exception {
String output = AwkTestSupport
.runCliInFreshJvm(
"system() child reads Jawk stdin",
"BEGIN { system(\"sort\") }",
"zulu\nalpha\n");
assertEquals("alpha\nzulu\n", output);
}

@Test
public void embeddedExecutionKeepsChildStandardInputClosed() throws Exception {
// A custom input stream cannot be lent to another OS process, so the
// child sees end of input at once and getline returns 0
AwkTestSupport
.cliTest("embedded cmd|getline child gets no standard input")
.script("BEGIN { n = (\"sort\" | getline line); print n \"[\" line \"]\" }")
.stdin("never seen by the child\n")
.expectLines("0[]")
.runAndAssert();
}
}