Skip to content

fix(exec): stop RunProcess from hanging on large stderr output - #882

Open
lbarrosby wants to merge 2 commits into
helm:mainfrom
lbarrosby:fix-runprocess-pipe
Open

lbarrosby wants to merge 2 commits into
helm:mainfrom
lbarrosby:fix-runprocess-pipe

Conversation

@lbarrosby

@lbarrosby lbarrosby commented Nov 6, 2026 •

Copy link
Copy Markdown

What this PR does / why we need it:

RunProcess wraps stdout and stderr in an io.MultiReader, so stderr is only read after stdout reaches EOF. A command that writes more than a pipe buffer (64 KiB on Linux) to stderr while stdout is still open blocks on the write, and ct hangs forever. helm install --debug, helm dependency build or kubectl logs with a chatty container can hit this.

Two more problems in the same function:

  • bufio.Scanner stops at lines longer than 64 KiB. The reader goroutine then exits and closes the pipes, and the command dies with SIGPIPE (failed waiting for process: signal: broken pipe).
  • cmd.Wait() can run before the goroutine has finished reading, and the os/exec docs say Wait must not be called before all reads from the pipe are done, so the end of the output can be lost.

This change passes the same writer as Stdout and Stderr, so os/exec uses one pipe for both and copies it to stdout as is. WaitDelay stops reading one second after the command exits, so background processes that inherited the pipe cannot keep ct waiting, and a failed write to stdout closes the read end so the command gets EPIPE instead of blocking. Write errors are returned.

pkg/exec/exec_test.go covers 1 MiB on stderr before stdout, a 1 MiB line, a command that leaves a background process holding the pipe (both with exit 0 and a failing exit status), and a broken stdout with 2 MiB of output. Each case runs under a timeout. go test -race ./... and golangci-lint pass.

Which issue this PR fixes: fixes #471

Special notes for your reviewer:

#471 was closed by the stale bot, but the cause is still in RunProcess.

AI disclosure: written with the help of an AI coding assistant, reviewed and tested by me.

RunProcess read stdout to EOF before reading stderr, so a command that
wrote more than a pipe buffer to stderr blocked forever. The scanner also
gave up on lines longer than 64KiB and closed the pipes, which killed the
command with SIGPIPE.

Write both streams to a single pipe, copy it to stdout as is, and only
call Wait once the copy is done.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>

@cpanato cpanato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This addresses a real deadlock: the added large-stderr test times out against the original implementation, and local go test ./... passes with the fix. However, I reproduced two additional hangs introduced by the new pipe lifecycle. Please address the inline findings and add regression coverage before merging.

Comment thread pkg/exec/exec.go Resolved
Comment thread pkg/exec/exec.go Resolved
Reading the pipe to EOF before calling Wait also waited for background
processes that inherited stdout and stderr, so a command that leaves a
service running kept ct blocked. If writing to stdout failed, the copy
stopped but the read end stayed open, and a command writing more than a
pipe buffer blocked before Wait could return.

Let exec own the pipe instead: passing the same non-file writer as
Stdout and Stderr still gives one shared pipe, WaitDelay stops reading
one second after the command exits, and a failed write closes the read
end so the command gets EPIPE. Write errors are returned to the caller.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@pull-request-size pull-request-size Bot added size/L and removed size/M labels Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sometimes when printing container logs ct will hang

2 participants